You'll build automated testing and evaluation for the company's AI agents and critical workflows, using AI to test AI. The role covers regression tests for APIs and CI/CD quality checks, as well as evaluating agent performance.
Strong test automation and programming skills in TypeScript or Python.
Hands-on experience with web apps, APIs, Git, and CI/CD.
A structured approach to judging AI output against clear, measurable criteria.
Ability to reproduce, isolate, and identify bugs, with attention to root cause.
Direct communication skills, including documenting decisions and sharing effective approaches.
Nice to have:
Experience with agentic browser testing or LLM evaluation frameworks.
Familiarity with Supabase, Vercel, or GCP.
Build and maintain API, integration, and end-to-end regression tests for critical workflows.
Evaluate AI agents on task completion, response quality, tool use, and permissions; measure how model, prompt, and context changes affect results.
Build quality checks into CI/CD pipelines and turn production defects into regression cases.
Use agents to explore workflows, generate tests, and reproduce bugs; check their findings.
Share effective tools and methods with the wider AI team and central QA department.
