Whisk

AI Quality & Evaluation Lead

Brak informacji o wynagrodzeniu

SeniorZdalnieFull-time

#450848·Dodano 19 dni temu·2

Źródło: Whisk

Aplikuj teraz

Tech Stack / Keywords

AILLMrubrictaxonomyevaluationnotebooksspreadsheets

Firma i stanowisko

At Samsung Food (known as Whisk), the company focuses on AI, nutrition, behavior change, and digital health, creating connected and personalized experiences for individuals and families. The team is remote-first with over 100 people across 30+ countries and is part of Samsung Electronics, combining startup agility with the scale of a large electronics company.

Wymagania

Experience running end-to-end evaluation loops on conversational or generated-text outputs, including error analysis, taxonomy creation, binary criteria development, and judge validation.
Backgrounds suitable for this role include conversation design, AI product quality, model behavior/policy, content design, human data operations, UX research with qualitative coding, RLHF, applied linguistics, or product management.
Ability to identify and document previously unnamed failure modes.
Comfortable acting as arbiter and documenting overrulings.
Understand rubric discovery is through grading, not upfront writing.
Ability to write annotation guidelines that require no interpretation.
Proficient working with notebooks and spreadsheets; no production coding required.
Domain expertise in nutrition, weight management, or behavior change is not required as it is provided in-house.

Obowiązki

Deliverables by Phase:

Author a failure taxonomy and create a golden set of hand-graded traces.
Develop a binary rubric with pass/fail criteria per taxonomy category and annotation guidelines.
Validate an LLM judge using prompts and provide weekly readouts on performance.
Create a written playbook covering all evaluation methods and perform handover testing.

Additional clarifications:

Not responsible for building evaluation infrastructure or dashboards.
Not accountable for functional or module-level evaluations or clinical safety validations.
Not performing high-volume annotation; volume work deferred to vendors or automated judge after rubric creation.

Whisk

4 aktywne oferty

Zobacz wszystkie oferty

Aplikuj teraz