Sii

Senior Automation Test Engineer – AI Systems (f/m/x)

Brak informacji o wynagrodzeniu

SeniorZdalnie·LublinFull-time·B2B·Umowa o pracę

#443316·Dodano miesiąc temu·10

Źródło: justjoin.it

Aplikuj teraz

Tech Stack / Keywords

TestingAIMachine Learning Large Language Models Python JavaScript TypeScript Node.js Angular

Podsumowanie oferty

You will work on building an AI-based operating system for healthcare marketing, focusing on quality, evaluation, and monitoring of complex AI systems. The role involves designing and scaling evaluation pipelines and monitoring systems for ML drift. Your work ensures the stability of the full-stack architecture and contributes to end-to-end testing strategies across frontend and backend components.

Wymagania

Minimum 5 years of experience in software testing or quality engineering, with at least 2 years in testing or evaluating AI/ML applications, LLM workflows, or agent systems
Proficiency in Python for backend/AI test scripting and familiarity with JavaScript/TypeScript in Node.js and Angular testing ecosystems
Experience with evaluation frameworks such as LangSmith, Langfuse, EvidentlyAI, Ragas, or MLflow and ability to select metrics for classification and generation quality problems
Knowledge of ML drift monitoring concepts, metric tracking, and observability tools like Datadog or Grafana
Strong system-level design skills and ability to build production-grade test and evaluation infrastructure from scratch
Advanced English language proficiency

Nice to have:

Familiarity with agent design patterns and decomposition into testing units
Experience working in regulated healthcare system environments

Obowiązki

Design, build, and maintain evaluation frameworks for large language models (LLM) and custom agent systems
Develop automated evaluation mechanisms, assertions, and test suites measuring agent accuracy, reasoning reliability, hallucination rates, and tool usage correctness
Simulate complex, multi-step agent workflows to discover edge cases in autonomous decision-making
Collaborate with engineering teams to establish robust monitoring frameworks for data drift, concept drift, and model performance degradation in production
Implement telemetry and alerting systems to track latency, token usage, cost efficiency, and result quality over time
Own end-to-end testing strategy across the stack, integrating Angular frontend, Node.js backend, and AI microservices in Python
Write clean, maintainable, and scalable test scripts in Python, JavaScript, and TypeScript
Conduct rigorous API, integration, and performance/load testing for high-throughput microservices
Integrate AI evaluation and testing systems into CI/CD pipelines for continuous quality enforcement before deployment
Define and track key quality indicators (KPIs) for software reliability and AI model behavior

Benefity

Great Place to Work certification
Stable financial situation
Contracts with global brands
Internal training center
Learning opportunities from many experts
Open and accessible management
Profit sharing
Passion Sponsorship Program
Regular integration events and trips
Comfortable and well-equipped offices
MySii application
Medical care

Udział w zyskach

Opieka zdrowotna

Inne informacje

Wyrażam zgodę na przetwarzanie moich danych osobowych zamieszczonych w niniejszym CV przez Sii sp. z o.o. z siedzibą przy Al. Niepodległości 69, 02-626 Warszawa („Sii”) dla celów procesów rekrutacyjnych prowadzonych przez Sii.

Sii

414 aktywnych ofert

Zobacz wszystkie oferty

Aplikuj teraz