QA Engineer (AI + backend)

160 - 170 PLN/ godz./ godz.B2BB2B (netto)

MidZdalnie·WarszawaFull-time·B2B

#460327·Dodano 6 dni temu·9

Źródło: justjoin.it

Aplikuj teraz

Tech Stack / Keywords

Technical QAAPI TestingPythonContract TestingSecurity-aware testingLLM EvaluationPrompt Engineering REST APIs Postman Git

Podsumowanie oferty

Own quality for a production AI coaching platform at a US-based health-tech company, focusing on LLM evaluation and backend APIs. Build evaluation and regression checks for AI responses, and test APIs and contracts using Python, LangGraph and Pact.

Wymagania

At least 3 years of experience in technical QA or test automation, including API testing on production systems.
Python test-code development and maintenance, including pytest and fixtures.
Strong HTTP and API fundamentals, including REST, JSON, status codes, headers, and authentication tokens.
Practical consumer-driven contract testing with Pact; this is a hard requirement.
Security-aware API testing, including authorization boundaries, token handling, and malformed or hostile input.
Ability to define acceptance criteria for non-deterministic output using properties and contracts rather than exact wording.
LLM evaluation experience, including datasets and golden sets, response quality metrics, and detecting regressions between prompt and model versions.
Prompt engineering skills, including measuring the effect of prompt changes and awareness of hallucinations and prompt injection.
Backend knowledge including HTTP, REST, JSON, tokens, logs, and database queries; ability to distinguish issues in the model, service code, or input data.
Daily use of Postman and Swagger/OpenAPI.
Git team workflow experience, including branches, pull requests, and reviews.
TestRail experience for test-case management and run tracking.
Mobile product experience with Android or iOS, including testing app-release timing and store builds.
Strong English for working with US colleagues.

Nice to have:

Langfuse or an equivalent such as LangSmith or Phoenix, including working with traces and attaching evaluations to them.
LangGraph testing, including testing individual nodes and full agent flows.
CI/CD experience running evaluations in pipelines and applying quality gates to pull requests.
Mobile automation exposure using Python with Appium.
Experience testing other AI- or LLM-driven features in production.
Experience taking over quality ownership of an existing product.

Obowiązki

Build and run LLM evaluations: curate datasets and golden sets, define response quality metrics, and detect quality regressions between prompt and model versions.
Define acceptance criteria for non-deterministic output using assertions on properties and contracts rather than exact wording.
Treat prompts as tested artifacts by measuring the effect of changes and probing for hallucinations and prompt injection.
Test REST endpoints, JSON payloads, authentication token flows, and error handling using Postman and Swagger/OpenAPI specifications.
Implement consumer-driven contract testing with Pact across the mobile app, gateway, and backend services.
Test authorization boundaries, token misuse, malformed input, and other API-level abuse cases.
Use logs, traces, and database queries to locate failures in the model, service code, or input data.
Help integrate evaluations into CI/CD pipelines and add quality gates to pull requests.
Map results to test cases in TestRail, track defects in Jira, and document findings.
Work in a shared workflow with the client's US-based engineering and QA teams and the Polish QA function.

Inne informacje

Work hours must overlap with US Mountain Time afternoons, approximately 14:00–18:00 Poland time.
Start as soon as possible.

apreel

194 aktywne oferty

Zobacz wszystkie oferty

Aplikuj teraz