Senior AI Engineer

The candidate will join a European health-technology team as a Senior AI Engineer responsible for ensuring the evaluation and quality assurance of a newly developed AI-driven module used by more than 2.5 million users across Europe. The module is certified as a Medical Device Class IIa and is built on a healthcare-specific large language model (LLM) platform.

• 6–8 years of hands-on software engineering experience, with at least 3 years focused on production AI and LLM applications.
• Extensive experience with Python as the primary language for building AI services, RAG pipelines, agents and evaluation tooling.
• Proven track record integrating LLM APIs and SDKs in production environments.
• Hands-on expertise in prompt engineering, RAG architectures and LLM orchestration frameworks.
• Demonstrable experience designing and operating LLM evaluation frameworks in production: accuracy metrics, hallucination detection, regression testing, ground truth dataset design and output monitoring.
• Familiarity working with .NET and C# at the integration level; ability to design clear Python-to-.NET API contracts and collaborate on a .NET codebase when required.
• Strong software engineering fundamentals: API design, automated testing, version control and CI/CD pipelines.
• Effective communication skills, able to engage clinicians and non-technical stakeholders on practical implications of model evaluation.
• Fluency in English (working language of the team).

Nice to have:

• Experience in healthtech, medtech or regulated industries.
• Background in MLOps or production AI monitoring.
• Familiarity with Medical Device Regulation (MDR), the EU AI Act or quality management for AI systems.
• Experience with clinical safety frameworks or human-in-the-loop systems.

• Develop and maintain production code for RAG pipelines, LLM orchestration, prompt engineering and agent workflows on the healthcare LLM platform.
• Design, implement and own the evaluation framework for model outputs, including accuracy metrics, hallucination detection, clinical safety checks and regression testing across releases.
• Create and maintain ground truth datasets for evaluation in collaboration with clinicians and content partners, ensuring data quality and traceability.
• Implement production monitoring and alerting for model outputs: drift detection, quality regressions and anomaly reporting.
• Produce evaluation evidence and traceability required for regulatory and quality processes, including planned AI and Quality Control certifications alongside existing ISO 27001 and ISO 27701 controls.
• Design robust API contracts between Python-based AI services and the .NET / Angular platform deployed on Azure; collaborate on integration and handoffs.
• Pair with the architecture-focused Senior AI Engineer on day-to-day implementation, share knowledge with the in-house engineering team and document evaluation practices.
• Engage with clinicians and non-technical stakeholders to translate clinical safety requirements into measurable evaluation criteria and test plans.
• Contribute to software engineering best practices: code review, automated testing, version control and CI/CD for AI services.