Software Engineering Evaluation Specialist

Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based, not permanent employment.

• 3+ years of production software development in one backend stack: Python, Go, Node.js, Java, or Rust.
• Fluency in Python and pytest, including fixtures, parametrize, monkeypatch, timeouts, and conftest.py.
• Experience authoring reproducible Dockerfiles with pinned dependencies and multi-stage builds.
• Comfort with Linux and Bash, including debugging with strace, lsof, journalctl, and advanced shell scripting.
• Experience with AI coding agents such as Claude Code, Cursor, or Roo Code, with examples of detecting agent errors.
• English proficiency at B2+ written level.

Not a fit:

• Data Science, ML, or Computer Vision engineers without backend output.
• Manual QA testers without automation or test authoring.
• Frontend-only, low-code/no-code, IT Support, or Business Analysts.
• Engineers who have never written pytest from scratch.
• Junior, intern, or assistant as most recent role.

Preferred qualifications:

• Domain depth in Security, System Administration (nginx, systemd, cron), Scientific Computing (NumPy, PyTorch, SciPy), DevOps, or Git internals.
• Modern Python tooling (uv, poetry, pyproject.toml).
• Coverage tooling (pytest-cov, coverage.py, gcov, llvm-cov, kcov).
• Fuzzing or property-based testing (Hypothesis).
• Prior contribution to agent-evaluation benchmarks or related frameworks.

• Invent a realistic developer scenario such as a real bug, broken ETL, or missing feature.
• Build a reproducible Docker environment with pinned dependencies.
• Write a pytest that verifies outcomes deterministically without leaking the fix.
• Write an instruction.md that reads like a Jira ticket for a developer.
• Write a reference solve.sh proving the task is solvable.
• Calibrate difficulty so AI agents solve the task 20–60% of the time.
• Iterate based on feedback from expert QA reviewers.
• Later review other authors' tasks as a QA reviewer.

Participation is project-based and not permanent employment. Candidates must submit CV in English indicating English proficiency. Tasks must be submitted by deadlines and meet acceptance criteria.