Senior Product Manager, AI Agents Testing

Zendesk AI Agents are fully autonomous agents that resolve customer issues end-to-end across messaging, email, and voice channels, handling millions of conversations for brands like Liberty London, Unity, and Motel Rocks. Zendesk powers billions of conversations and offers a hybrid way of working with offices worldwide.

• Several years of product management experience, including 2+ years building for non-technical users in complex technical domains in B2B SaaS
• Experience shipping AI/ML products with focus on evaluation and reliability
• Understanding of limitations of traditional testing for LLM-based systems and opinions on effective approaches
• Ability to ship platform capabilities through user-facing product surfaces
• Experience integrating acquired or adjacent products into a unified experience
• Track record coordinating across multiple engineering teams and departments

Bonus Qualifications:

• Experience building simulation, synthetic data, or automated testing products
• Background in conversational AI, chatbot platforms, or customer service technology
• Familiarity with LLM evaluation approaches including human-in-the-loop scoring and automated rubrics
• Experience with experimentation infrastructure such as A/B testing and feature flagging at scale
• Experience turning internal prototypes into customer-facing products

• Own product strategy and roadmap for AI agent testing including simulation, quality scoring, experimentation, regression detection, and conversation tracing
• Ship testing as an integrated experience embedded in the builder and deployment flow
• Define end-to-end simulation: scenario generation from real conversation patterns, automated pass/fail evaluation, and detailed results
• Build experimentation layer with A/B testing, staged rollouts, and safe iteration on tone and resolution strategies
• Design a pre-publish readiness gate providing admins a quantified view of risk before deployment
• Partner with ML, QA, and platform teams on scoring methodology, simulation infrastructure, and tracing architecture
• Make the testing platform usable by non-technical admins such as CX managers, bot builders, and operations leads

Zendesk is an equal opportunity employer and complies with applicable laws regarding nondiscrimination. Applicants based in the United States can access EEO rights information. Reasonable accommodations are available for applicants with disabilities.