Automation QA Engineer

Ciklum is a custom product engineering company with over 4,000 developers, consultants, analysts, and product owners. The company supports multinational organizations and scaling startups by engineering technology solutions. It has delivery centers in Wrocław and Gdańsk with 300+ professionals in Poland.

• 3+ years of experience in backend test automation, data quality engineering, or Python software development within structured delivery environments
• Mastery of Python for backend testing, scripting, and data manipulation; traditional UI automation tools like Selenium or Cypress are not applicable
• Hands-on experience or deep technical familiarity with LLM validation frameworks such as RAGAS
• Practical experience configuring automated script execution, regression test triggers, and quality gates in Git-based pipelines, preferably Azure DevOps
• Basic functional understanding of relational databases (PostgreSQL), data profiling techniques, and search indexing concepts
• Must be physically located in Poland or Eastern Europe to align with the embedded co-delivery squad

• Build and run automated validation test suites using Python frameworks like RAGAS to score pipeline faithfulness, answer relevance, and context precision
• Integrate automated LLM evaluation scorecards into Azure DevOps pipelines to prevent regression in code-generation quality
• Develop automated checks to monitor documentation ingestion pipelines, chunking consistency, and vector store retrieval accuracy
• Generate programmatic quality scorecards and regression reports for project gate approval milestones
• Collaborate with senior and mid-level ML engineers to isolate, reproduce, and patch chunking or retrieval flaws exposed by validation metrics

Must be physically located in Poland or Eastern Europe to remain aligned with the embedded co-delivery squad.