Mid-Level AI & Automation QA Engineer – AI-Driven Testing

The role is for a client leading in AI-powered features in conversational AI, productivity tools, and rating systems, based in Porto, Portugal with a hybrid work model.

• 3+ years experience in QA/SDET roles with a focus on automation testing.
• Hands-on experience with automated test frameworks: Playwright, Cypress, Selenium.
• Proficiency in scripting: Python, JavaScript, TypeScript.
• Understanding AI-specific testing challenges like prompt regression and hallucination detection.
• Experience designing evaluation datasets and using AI observability/tracing tools.
• Knowledge of AI fairness, bias mitigation, and compliance testing.
• Familiarity with CI/CD pipeline integration for automated testing.
• Strong root-cause analysis skills for both code and AI outputs.

Nice to have:

• Experience with AI-assisted test generation or self-healing scripts.
• Knowledge of LLM evaluation approaches and AI debugging tools.
• Certifications related to QA or AI safety.

• Develop and maintain automated test frameworks using tools like Playwright, Cypress, or Selenium.
• Script custom testing solutions leveraging Python or JavaScript/TypeScript.
• Design and implement evaluation datasets and automated pipelines, including LLM-based judgment approaches.
• Analyze AI outputs to detect hallucinations, errors, and biases.
• Integrate AI evaluation and functional tests into CI/CD pipelines.
• Utilize AI observability and tracing tools for debugging production models.
• Craft and execute test cases for conversational AI, including dialogue flows and adversarial prompts.
• Collaborate on testing to ensure AI fairness, bias mitigation, and compliance.

Only candidates with an existing legal right to work in the European Union will be considered for this role.