Software Engineering Evaluation Specialist

Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based, not permanent employment.

• Minimum 3 years of production software development in one backend stack: Python, Go, Node.js, Java, or Rust; depth in one stack preferred.
• Fluency in Python and pytest, including fixtures, parametrize, monkeypatch, timeouts, conftest.py.
• Ability to author Dockerfiles: reproducibility, pinned dependencies, multi-stage builds, non-root users.
• Comfortable with Linux and Bash for debugging inside containers (tools like strace, lsof, journalctl).
• Experience with AI coding agents such as Claude Code, Cursor, Roo Code, on non-trivial work.
• Written English proficiency at B2+ level.

Not a fit:

• Data Science, ML, or Computer Vision engineers without backend output.
• Manual QA testers without automation or test authoring.
• Frontend-only, low-code/no-code, IT Support, or Business Analysts.
• Candidates who have never written pytest from scratch.
• Junior, intern, or assistant as the most recent role.

Preferred qualifications:

• Depth in Security, System Administration (nginx, systemd, cron), Scientific Computing (NumPy, PyTorch, SciPy), DevOps, or Git internals.
• Modern Python tooling: uv, poetry, pyproject.toml.
• Coverage tooling: pytest-cov, coverage.py, gcov, llvm-cov, kcov.
• Fuzzing or property-based testing (Hypothesis).
• Prior contributions to agent-evaluation benchmarks or related frameworks.

• Invent realistic developer scenarios such as real bugs, broken ETL, or missing features.
• Build reproducible Docker environments with pinned dependencies.
• Write pytest tests that verify outcomes deterministically without leaking the fix.
• Compose instruction.md files resembling Jira tickets developers would receive.
• Develop reference solve.sh scripts proving tasks are solvable.
• Calibrate difficulty for state-of-the-art AI agents to solve tasks 20–60% of the time.
• Iterate based on expert QA reviewers' feedback.
• Review other authors’ tasks as a QA reviewer (later).

Please submit your CV in English and indicate your level of English proficiency. Participation is project-based, not permanent employment. Remote work allowed. Candidates must demonstrate written English at B2+ level.