Agentic AI Technical Test Lead | Poland

Deloitte offers roles working on innovative AI engagements for enterprise clients, collaborating with AI, data, cloud, cybersecurity, and consulting experts. The company provides career paths toward AI Quality Engineering Lead, AI Assurance Architect, or AI Testing Offering Lead and encourages building thought leadership and reusable AI testing assets.

• Strong background in Quality Engineering, test automation, AI/ML testing, software testing, or technical test leadership
• Practical understanding of generative AI, LLM-based applications, RAG, and Agentic AI solutions
• Ability to define test strategies, quality gates, and evaluation approaches for AI solutions
• Experience with AI quality dimensions such as accuracy, precision/recall, relevance, groundedness, hallucination risk, robustness, consistency, safety, and compliance
• Ability to design evaluations for prompts, AI-generated responses, agent workflows, tool-calling, multi-step reasoning, RAG pipelines, and human-in-the-loop review
• Good technical understanding of APIs, integrations, cloud platforms, CI/CD, and solution architecture
• Hands-on experience with Python or similar scripting/programming language
• Excellent English, both written and spoken
• Strong presentation and stakeholder management skills
• Consulting mindset, ownership attitude, and ability to support sales/pre-sales activities
• Passion for AI and willingness to actively share knowledge across teams
• Residence in the European Union

AI Testing Strategy & Standards:

• Define reusable testing standards for Agentic AI solutions
• Create evaluation frameworks for prompts, RAG, agent workflows, and tool-calling
• Define quality gates, acceptance criteria, test metrics, and reporting templates
• Establish approaches for accuracy, precision, hallucination, robustness, and safety evaluation
• Build reusable assets, accelerators, and best practices for AI testing

AI Evaluation & Execution:

• Lead test design and execution for AI agents and LLM-based applications
• Create golden datasets, evaluation scenarios, and adversarial test cases
• Validate agent reasoning, tool usage, API interactions, and workflow orchestration
• Support automated and human-in-the-loop evaluation processes
• Analyze results and provide clear recommendations to technical and business stakeholders

Technical Solution & Delivery Support:

• Design technical testing solutions for client engagements
• Embed evaluation into CI/CD, DevOps, MLOps, or LLMOps pipelines
• Support observability, monitoring, feedback loops, and production quality controls
• Work with architects, AI engineers, data engineers, and business teams
• Identify quality risks and propose practical mitigation actions

Consulting & Business Development:

• Support pre-sales, proposals, estimations, and client presentations
• Prepare technical materials explaining AI testing approaches
• Facilitate workshops and discovery sessions
• Act as a trusted advisor on AI quality, reliability, and risk
• Promote AI testing knowledge internally and externally

Residence in the European Union is required for this role.