Senior Automation Test Engineer – AI Systems (f/m/x)

• Minimum 5 years of experience in software testing or quality engineering, with at least 2 years in testing or evaluating AI/ML applications, LLM workflows, or agent systems
• Proficiency in Python for backend/AI test scripting and familiarity with JavaScript/TypeScript in Node.js and Angular testing ecosystems
• Experience with evaluation frameworks such as LangSmith, Langfuse, EvidentlyAI, Ragas, or MLflow and ability to select metrics for classification and generation quality problems
• Knowledge of ML drift monitoring concepts, metric tracking, and observability tools like Datadog or Grafana
• Strong system-level design skills and ability to build production-grade test and evaluation infrastructure from scratch
• Advanced English language proficiency

Nice to have:

• Familiarity with agent design patterns and decomposition into testing units
• Experience working in regulated healthcare system environments

• Design, build, and maintain evaluation frameworks for large language models (LLM) and custom agent systems
• Develop automated evaluation mechanisms, assertions, and test suites measuring agent accuracy, reasoning reliability, hallucination rates, and tool usage correctness
• Simulate complex, multi-step agent workflows to discover edge cases in autonomous decision-making
• Collaborate with engineering teams to establish robust monitoring frameworks for data drift, concept drift, and model performance degradation in production
• Implement telemetry and alerting systems to track latency, token usage, cost efficiency, and result quality over time
• Own end-to-end testing strategy across the stack, integrating Angular frontend, Node.js backend, and AI microservices in Python
• Write clean, maintainable, and scalable test scripts in Python, JavaScript, and TypeScript
• Conduct rigorous API, integration, and performance/load testing for high-throughput microservices
• Integrate AI evaluation and testing systems into CI/CD pipelines for continuous quality enforcement before deployment
• Define and track key quality indicators (KPIs) for software reliability and AI model behavior

Wyrażam zgodę na przetwarzanie moich danych osobowych zamieszczonych w niniejszym CV przez Sii sp. z o.o. z siedzibą przy Al. Niepodległości 69, 02-626 Warszawa („Sii”) dla celów procesów rekrutacyjnych prowadzonych przez Sii.