AI Evaluation Engineer (Python, QA or Security)

Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems.

• At least 5 years of experience in software development.
• Experience with Python (FastAPI), JavaScript, TypeScript (React), Docker, Postgres, Kafka, and Redis.
• Experience writing functional and integration tests.
• English language proficiency at B2 level or higher.

• Build realistic developer environments resembling a virtual company with codebase, infrastructure, and context such as tickets, docs, and conversations.
• Design tasks from intermediate states of these environments including crafting prompts, defining success criteria, and ensuring solvability by an AI agent.
• Write tests that validate agent solutions, accepting all valid approaches and rejecting incorrect ones with balanced strictness.
• Iterate on tasks and tests based on QA feedback by reviewing agent solutions, analyzing failures, and refining evaluations for fairness and robustness.

Participation is project-based and not permanent employment. Candidates must submit CVs in English and indicate English proficiency. Position is remote and open for specialists located in Poland.