Senior AI QA Test Automation Engineer – CX Agent

XM is a leading international FinTech company, established in 2009, with over 1400 employees globally. Headquartered in Cyprus, XM operates offices in Greece, UK, UAE, USA, South Africa, and Uruguay. XM has received Platinum accreditation from Investors In People and consistent recognition as one of the top Best Workplaces™.

• BSc/MSc in Computer Science, AI, or related discipline
• 6+ years hands-on experience in QA/Test Automation with strong automation framework design and maintenance
• 1+ years experience applying AI/ML in software testing or QA process improvement with production workflows
• Strong Python skills and experience in Python/AWS-centric environments; Java and/or TypeScript is a plus
• Hands-on experience with LLM evaluation techniques including LLM-as-judge, human-in-the-loop evaluation, RAG, and multi-agent orchestration
• Practical experience with DeepEval or comparable frameworks and familiarity with agent-orchestration SDKs like Strands Agents
• Knowledge of LLM tooling, vector databases, and MLOps pipelines
• Experience integrating AI tooling into enterprise GitLab CI/CD and working with containerized cloud-native environments such as Docker and Kubernetes/EKS
• Strong communication skills to influence technical decisions and drive engineering standards

Nice to have:

• Experience with autonomous QA agents or agentic orchestration frameworks
• Experience with LLM observability tools such as LangFuse, LangSmith, or Arize
• Knowledge of AI ethics, fairness, and bias detection for guardrails and model validation
• Experience with gRPC, WebSockets, and/or HTTP/2
• Experience with AWS Bedrock, and familiarity with GCP Vertex AI and/or Azure AI

• Act as the primary technical enabler for QA, building scalable AI/ML frameworks, libraries, and tooling for the CX Agent Suite and beyond
• Design and build AI evaluation pipelines assessing multi-agent responses for accuracy, relevance, tone, hallucination rate, safety/guardrail compliance, and task completion
• Evaluate and prototype Strands Agents or comparable agent-orchestration frameworks for autonomous QA agents and lead adoption decisions
• Collaborate with QA, Data Science, and Engineering to integrate AI testing into the GitLab CI/CD pipeline and Terraform-provisioned, EKS/AgentCore-hosted services
• Build resilient and adaptive automation to respond to changes in agent behavior and Bedrock Guardrails configuration
• Develop data-driven quality analytics including root-cause analysis, quality trends, and intelligent test prioritization leveraging OpenTelemetry, CloudWatch, X-Ray, and LangFuse
• Lead research into emerging AI testing methodologies and mentor engineers through code reviews, workshops, and architectural guidance