**Role & seniority: ** Automation QA Engineer (3+ years experience); mid-level

**Stack/tools: ** Python, PyTest; AI/LLM testing with non-deterministic text, confidence scoring, NLI assertion; test automation frameworks; Neo4j (Cypher), Qdrant; Docker; Git; REST API automation; CI/CD; secure zero data egress tenant; evaluation “golden datasets”

**Top 3 responsibilities: **

Design/run automated test suites validating LLM-generated documentation against code truth with NLI accuracy thresholds
Validate backend/data pipeline correctness (Neo4j state/schema, Qdrant vector correctness, ETL/batch transformations)
Create regression/evaluation frameworks to detect output quality drift, hallucination rates, and pipeline performance across releases
Must-have skills:
- Strong AQA with backend/data pipelines or AI/LLM applications (3+ yrs)
- Direct experience testing LLM outputs (non-deterministic evaluation, scoring, NLI logic)
- Direct experience in ETL/data pipeline testing (schema integrity, batch processing correctness)
- Proficiency in Python + PyTest
- Ability to write DB queries (Neo4j/Cypher, Qdrant/SQL)
- CI/CD integration, Docker, Git, REST API automation
- English proficiency (B2+/C1)
Nice-to-haves:
- Experience with enterprise AI documentation pipelines and evaluation baselines/golden datasets
- Familiarity with secure/regulated delivery requirements (e.g., SBOM/source traceability, EU CRA)
Location & work type: Poland; fu

Full Description

Ciklum is looking for an Automation QA Engineer to join our team full-time in Poland.

We are a custom product engineering company that supports both multinational organizations and scaling startups to solve their most complex business challenges. With a global team of over 4,000 highly skilled developers, consultants, analysts and product owners, we engineer technology that redefines industries and shapes the way people live.

About the role

As an Automation QA Engineer, become a part of a cross-functional development team engineering experiences of tomorrow. The Legacy Code Semantic Documentation Project is a 26-week enterprise initiative for a global industrial automation leader. The primary objective is to engineer an automated, AI-assisted pipeline to generate structured, system-level Markdown documentation directly from an undocumented ~400K LOC codebase (spanning IEC 61131-3 languages and ANSI C/C++). Operating within a dedicated, zero-data-egress secure tenant, the technical architecture combines Tree-sitter AST parsing, Neo4j knowledge graphs, Qdrant vector search, and self-hosted open-weight LLMs (Llama 3.1 / Mixtral family) with NLI-based validation. Delivery is structured across 7 work packages

executing over a 6 month period, governed by strict contractual KPIs: ≥95% code coverage, ≥92% NLI-verified factual precision, and ≥90% SME validation acceptance. All outputs align with EU Cyber Resilience Act (EU CRA) requirements for SBOM and source-level traceability.

Responsibilities

*** AI Output Quality Assurance:** Design and run automated test suites to validate non-deterministic LLM-generated documentation against code source truth and strict NLI accuracy thresholds

*** Data Pipeline Validation:** Verify state consistency, schema integrity, and data transformation accuracy across batch executions in Neo4j knowledge graphs and Qdrant vector databases

*** Regression & Evaluation Testing:** Build automated test frameworks to track output quality drift, hallucination rates, and pipeline performance across code releases

*** Test Dataset Management:** Create and maintain evaluation baselines ("golden datasets") to continuously benchmark LLM prompt outputs and retrieval accuracy

*** Cross-Functional Collaboration:** Partner closely with Senior AI/ML Engineers and Data Engineers to define automated acceptance criteria, defect logging, and pipeline health checks

Requirements

*** Professional Experience:** 3+ years in Automation Quality Assurance (AQA) with a heavy focus on backend systems, complex data pipelines, or AI-driven applications

* Mandatory Domain Experience

- Direct experience testing AI/LLM outputs (evaluating non-deterministic
- generated text, confidence scoring, and NLI assertion logic)
- Direct experience in data pipeline & ETL testing (verifying batch processing,
- schema integrity, and data transformations)

*** Automation & Scripting:** Advanced proficiency in Python and test frameworks such as PyTest

*** Database & Querying Skills:** Hands-on experience writing queries to inspect graph databases (Neo4j/Cypher), vector search stores (Qdrant), or SQL databases

*** Engineering Standards:** Proficiency with Docker, Git, REST API automation, and integrating tests into CI/CD pipelines

*** Language:** Professional proficiency in English (B2+/C1)

What’s in it for you?

*** Strong community:** Work alongside top professionals in a friendly, open-door environment

*** Growth focus:** Take on large-scale projects with a global impact and expand your expertise

*** Tailored learning:** Boost your skills with internal events (meetups, conferences, workshops), Udemy access, language courses, and company-paid certifications

*** Endless opportunities:** Explore diverse domains through internal mobility, finding the best fit to gain hands-on experience with cutting-edge technologies

*** Flexibility:** Enjoy flexibility – full remote working possibilities

*** Care:** We've got you covered with company-paid premium medical package through Luxmed

*** Benefits:** Access the MyBenefit cafeteria platform, allowing you to choose perks that best suit your lifestyle and needs

About us

At Ciklum, we are always exploring innovations, empowering each other to achieve more, and engineering solutions that matter. With us, you’ll work with cutting-edge technologies, contribute to impactful projects, and be part of a One Team culture that values collaboration and progress.

With delivery centers in Wrocław and Gdańsk, our 300+ professionals in Poland drive forward-thinking solutions for global clients. Join a community where collaboration sparks innovation—and your impact reaches millions.

Explore, empower, engineer with Ciklum!

Interested already? We would love to get to know you! Submit your application. We can’t wait to see you at Ciklum.

#LI-JW1