**Role & seniority: ** Full-time AI Quality Assurance Expert (mid–senior; 5–10 years in QA/test validation)

**Location & work type: ** Remote, must be based in Portugal

**Stack/tools (explicit/implicit): **

Python (review/assessment of validation & test automation code)
QA/test automation frameworks (unspecified)
API testing
MLOps
Observability/monitoring for AI systems
Defect tracking & audit-ready evidence management
AI evaluation artifacts: evaluation datasets, benchmarks, prompt test suites, regression packs

**Top 3 responsibilities: **

Define and execute QA strategies for AI-powered apps (quality gates, automated tests, regression packs).
Validate AI/LLM/agentic systems (robustness, simulation-based validation, non-deterministic quality monitoring).
Manage AI-specific risk assurance (hallucinations, drift, bias, prompt injection) and produce audit-ready defect/test evidence.

**Must-have skills: **

AI/ML/LLM or data-driven systems QA experience
Strong Python for code review of validation/test automation
Knowledge of test automation, API testing, MLOps, observability/monitoring
Familiarity with AI validation (model evaluation, LLM testing, agentic systems)
Documentation/evidence discipline; analytical detail; clear communication

**Nice-to-haves (implied): **

Experience across **prompt test suite desig

Full Description

We are looking for a full-time AI Quality Assurance Expert to ensure the quality, reliability, and robustness of AI-enabled solutions. This role combines traditional software QA with AI-specific validation, supporting the development of machine learning models, LLM-based applications, agentic systems, and other intelligent products.

Work Model: Remote (must be based in Portugal)

Responsibilities

Define and execute QA strategies for AI-powered applications. Design quality gates, evaluation datasets, benchmark scenarios, prompt test suites, regression packs, and automated tests. Validate AI models, LLMs, and agentic systems. Assess and mitigate risks such as hallucinations, model drift, bias, and prompt injection. Review Python-based validation and test automation code. Establish monitoring, observability, and quality metrics for AI systems. Manage defect tracking, test documentation, and audit-ready evidence. Translate technical findings into business impact assessments and recommendations. Collaborate with engineering, data science, product, compliance, cybersecurity, and business teams.

Requirements

5–10 years of experience in Software QA, Test Automation, or Digital Product Validation. Practical experience with AI, Machine Learning, LLM-based applications, or data-driven systems. Strong Python skills, with the ability to review and assess test and validation code. Good understanding of test automation frameworks, API testing, MLOps, observability, monitoring, and defect management. Experience with AI validation, including model evaluation, LLM testing, agentic systems, robustness testing, simulation-based validation, and quality monitoring of non-deterministic systems. Knowledge of AI-specific risks such as hallucinations, drift, bias, and prompt injection. Strong analytical skills, attention to detail, and excellent communication abilities. Comfortable working in agile, international, and cross-functional environments. Strong documentation and evidence management discipline.