As a Senior Automation QA Engineer, you will ensure the end-to-end quality of a multi-tenant, serverless AWS platform for building, deploying and running AI agents. You will define test strategies and build API and UI automation, including with Playwright, Python or TypeScript.
5+ years of experience in software quality engineering.
Experience defining a test strategy for a web platform or SaaS product.
Experience setting up and owning a test automation repository, including its structure, conventions, CI integration, code reviews and long-term maintenance, using Python or TypeScript.
Hands-on experience automating REST API tests, for example with pytest and httpx, or Playwright API testing.
Experience with schema validation against OpenAPI; contract testing is a plus.
Experience testing asynchronous flows, including polling, state machines, queues and WebSockets.
Experience testing authentication and access control, including OAuth2/OIDC, JWT, role- and tenant-based access, and negative cases.
Proven experience with Playwright on React/SPA applications, including keeping a large suite stable and flaky tests measured and under control.
Experience building quality gates in GitLab CI or a similar tool with Docker.
Experience testing cloud-hosted applications and debugging through logs, metrics and traces.
Understanding of how LLM-based agents behave, including tools, memory and non-determinism.
Daily use of AI assistants such as Claude Code or GitHub Copilot for writing tests, analysing failures and documenting; ability to critically review AI-generated code.
Strong collaboration and reporting skills, ability to work in cross-functional and international teams, and strong analytical and problem-solving skills with attention to detail.
Nice to have:
Experience with AWS, including Lambda, API Gateway, DynamoDB, S3, CloudWatch and Bedrock AgentCore.
Experience with Strands or other agent frameworks, such as LangGraph or CrewAI.
Familiarity with LLM evaluation tools, such as DeepEval, Promptfoo or Ragas.
Experience with MCP/A2A servers and protocols.
Knowledge of vector-based retrieval systems.
Experience in monitoring and observability.
Experience with full-stack application testing.
Working knowledge of the OWASP Top 10 for web and API security, and hands-on penetration testing experience, such as with Burp Suite or OWASP ZAP.
Knowledge of GxP or regulated environments.
Test Strategy:
Define the test strategy and test pyramid with developers and platform engineers, and keep them up to date as the platform grows.
Test Repository Ownership:
Set up and own the automation repository, including its structure, conventions, shared fixtures, code review and reporting.
Keep the repository simple enough for developers to add tests themselves.
API Test Automation:
Build functional, schema, negative and regression suites for the REST API.
Cover long-running asynchronous flows, including agent deployments, image builds, provisioning development environments and WebSocket log streaming.
UI Test Automation:
Build a stable Playwright end-to-end suite for critical user journeys: workspaces, agent deployment, marketplace, evaluation, monitoring and administration.
Agent Integration & Smoke Testing:
Deploy supported agent templates through the platform.
Check endpoint responses, authentication enforcement, logs and traces, MCP tool calls and memory persistence.
CI/CD & Test Environments:
Run test suites in CI as merge-request gates, nightly regression runs and post-deployment smoke tests.
Automate the setup and cleanup of short-lived test workspaces and test data in AWS.
Collaboration and Troubleshooting:
Work with developers, AI engineers and platform teams to identify automation opportunities and improve software quality.
Conduct root cause analysis for identified issues and collaborate with development teams on resolution.