Senior Research Engineer (Agentic Behavior)

JetBrains is a software company founded in 2000 focusing on developer tools. The Kotlin AI Value Stream team builds evaluation infrastructure, error analysis tools, and post-training pipelines that improve AI agents' behavior on Kotlin developer tasks across multiple platforms including Android, Kotlin Multiplatform, server-side, web, and desktop.

• Hands-on experience building evaluation or analysis pipelines for LLMs or AI coding agents.
• Strong Python engineering skills with at least three years of experience writing clean and maintainable code.
• Experience with data analysis at scale using SQL and Athena, including building data pipelines and statistical analysis.
• Ability to own projects end to end from problem identification to shipping fixes.
• Product-aware mindset understanding developers' needs and failure modes.
• Familiarity with Kotlin or strong willingness to develop deep expertise.

Nice to have:

• Experience with post-training LLMs: SFT, RLHF, DPO, GRPO.
• Experience with modern deep learning frameworks like PyTorch and LLM training stacks.
• Experience in AI agent development including tool-using agents and agentic frameworks.
• Familiarity with evaluation tools such as Inspect AI, Promptfoo, or LM-evaluation-harness.
• Experience with experiment tracking tools like MLflow.
• Knowledge of the Kotlin ecosystem including Android, Gradle, Spring, and KMP.
• Experience contributing to open-source projects, especially benchmarks or evaluation tools.

• Design and implement tooling to capture, classify, and analyze errors made by AI coding agents generating Kotlin code.
• Build observability pipelines over agentic traces from various coding agents and JetBrains IDEs.
• Design, implement, and maintain evaluation pipelines measuring Kotlin code generation quality across multiple dimensions.
• Build simulation environments to measure coding agents on realistic Kotlin developer tasks.
• Own evaluation infrastructure including metrics, experiment tracking, automated regression checks, and benchmarking.
• Experiment with post-training techniques like SFT, DPO, and GRPO to improve model handling of Kotlin patterns.
• Investigate context engineering approaches such as compiler feedback loops and Kotlin LSP integration.
• Run experiments including A/B tests and benchmark analyses on real codebases.
• Collaborate with model providers to incorporate Kotlin-specific improvements.
• Design and maintain open-source benchmarks measuring AI coding agent performance on Kotlin tasks.
• Create diverse task datasets covering server-side, multiplatform projects, build systems, Android, and libraries.
• Maintain and update benchmarks to ensure relevance and challenge as models evolve.