Senior QA / ML Tester

• 5+ years of production experience in QA automation or ML/AI system testing
• Proven experience testing AI/LLM systems, including evaluation pipelines and model behavior validation
• Proficiency in Python test automation, including pytest, fixtures, parametrize, and mocking (unittest.mock, moto)
• Knowledge of AWS AgentCore Evaluation API covering on-demand and online modes
• Familiarity with OpenTelemetry/ADOT for trace-based evaluation and validation
• Skills in REST API testing with authentication (SigV4, bearer tokens)
• Experience with CI/CD integration using GitHub Actions, Jenkins, or equivalent
• Background in test data management, synthetic trace generation, and defect lifecycle management using Jira
• Understanding of LLM/agent evaluation concepts like correctness scoring and evaluator chaining
• Ability to work independently, manage tasks, report status, and proactively escalate blockers
• Strong analytical skills for defining test scenarios from evolving specifications
• English B2+ language proficiency for daily collaboration

Nice to have:

• Hands-on experience with AWS AgentCore Evaluation API or AWS Bedrock testing
• Experience testing OpenTelemetry/distributed tracing pipelines
• Familiarity with multi-evaluator execution patterns and correctness strategies
• Experience writing technical feasibility reports
• Knowledge of agentic AI frameworks (LangGraph, Strands Agents)
• ISTQB CT-AI certification or equivalent AI testing qualification
• Experience with AI Ready/Practitioner practices at EPAM including prompt engineering

• Design and implement a functional test suite for the AWS AgentCore Evaluation API using pytest, covering session pair validation, multi-evaluator correctness, and edge cases
• Develop and maintain integration tests for on-demand evaluation mode integrated into the CI/CD pipeline
• Validate online mode sampling accuracy, define test scenarios and acceptance criteria, and report deviations
• Conduct and document feasibility assessments for non-AgentCore runtime evaluations, analyzing alternative runtimes
• Test OpenTelemetry trace-based evaluation inputs and validate ADOT trace ingestion and structure
• Collaborate with platform engineers to clarify evaluation contracts and reproduce defects
• Maintain test documentation in tools like Confluence and Jira
• Participate in Agile ceremonies including sprint planning, daily standups, demos, and retrospectives
• Contribute to engineering excellence practices such as code review, CI/CD integration, and coverage reporting

EPAM is a global leader in AI transformation engineering and integrated consulting, serving Forbes Global 2000 companies and startups. With over thirty years in software and platform engineering, EPAM empowers clients to become AI-native enterprises driving measurable innovation and digital value.