Lead AI Testing Engineer

• 8+ years of experience in software development, testing automation and DevOps
• Proficiency in Python test automation using pytest or equivalent, scripting and AI/ML library integration
• Expertise in testing Knowledge Graph or Data Ontology solutions using OWL2, RDF and SPARQL
• Skills in SHACL validation and graph databases such as GraphDB, including ontology validation, entity/relationship integrity, reasoner consistency checks and graph query correctness
• Proven ability to define and drive QA strategy independently for complex data-heavy systems, including building automation frameworks and implementing quality gates
• Skills in data validation using SQL and graph query languages such as SPARQL and Cypher, along with understanding of deterministic vs probabilistic systems
• Background in backend and API testing (REST) covering data validation, integration and E2E testing
• Proficiency in frontend web application test automation using Playwright or equivalent Python-compatible framework
• Hands-on experience using coding agents and agentic development daily
• Demonstrated experience testing and evaluating RAG-based Gen AI / LLM applications including grounding, answer accuracy and hallucination/determinism checks
• Applied knowledge of Gen AI / LLM evaluation frameworks and metrics such as precision, recall, criteria recall and efficiency, with proven ability to automate Gen AI/RAG evaluation at scale
• Experience in Agile delivery using Git and Azure DevOps or JIRA
• English proficiency at B2 level or higher

Nice to have:

• Skills in semantic search testing and evaluation
• Familiarity with Vector Database integration and retrieval validation
• Background in data science covering ML concepts, data pipelines or data engineering collaboration
• Proficiency in API tooling such as Postman and Bruno
• Experience with quality-gate implementation in CI/CD pipelines such as Azure DevOps or Jenkins

• Design and automate evaluation of RAG-based and LLM-driven features including grounding, answer accuracy, determinism/reproducibility, precision, recall and hallucination rate
• Build test harnesses to scale evaluation beyond human-in-the-loop processes
• Create data-driven test suites for underwriting rules and validate rule execution via SPARQL evaluator, arithmetic/threshold logic and multi-condition scenarios
• Verify ontology schema correctness and instance accuracy against source data, perform reasoner consistency checks and SHACL validation
• Define and drive end-to-end QA strategy covering data ingestion, ontology/graph layer, rule evaluation engine, AI/LLM layer and integration with underwriting solutions
• Establish quality gates, acceptance criteria and test coverage models
• Maintain automation test suites across knowledge graph, data ontology layer, backend services and frontend layers
• Validate end-to-end flows across ontology, rule engine and decision output, perform contract testing and cross-system data consistency validation
• Integrate test suites into CI/CD pipelines and define release quality gates
• Champion automation-first and shift-left quality practices
• Leverage agentic AI and Gen AI tooling in testing and framework development

EPAM is a leading global provider of digital platform engineering and development services. We are committed to having a positive impact on our customers, our employees, and our communities. We embrace a dynamic and inclusive culture. Here you will collaborate with multi-national teams, contribute to a myriad of innovative projects that deliver the most creative and cutting-edge solutions, and have an opportunity to continuously learn and grow. No matter where you are located, you will join a dedicated, creative, and diverse community that will help you discover your fullest potential.