Senior Software Engineer - AI Model Evaluation

• Minimum 5 years of experience in software development
• Proficient with core stack technologies including Python (with FastAPI), JavaScript, TypeScript (with React), Docker, Postgres, Kafka, Redis
• Experience writing functional and integration tests
• English proficiency at B2 level or higher

• Build realistic developer environments, including codebase, infrastructure, and context such as tickets, docs, and conversations that form a believable development history
• Design tasks from intermediate states of these environments by crafting prompts and defining success criteria ensuring tasks are solvable by AI agents
• Write functional and integration tests to verify AI agent solutions, accepting all valid approaches and rejecting incorrect ones
• Iterate on tasks and tests based on QA feedback by reviewing agent solutions, analyzing failures, and refining evaluations for fairness and robustness

Applicants must submit CVs in English and indicate their level of English proficiency. This opportunity is project-based, not permanent employment. Tasks must meet acceptance criteria and deadlines for payment.