Role & seniority: AI QA Specialist (mid/senior level)

Stack/tools: AI/LLM/conversational systems; QA for non-deterministic outputs; JSON schemas; datasets; CI/CD pipelines; observability/monitoring tooling; scripting (Python preferred)

Top 3 responsibilities

Define, evolve, and maintain evaluation strategies for AI, LLM, and conversational systems
Design and manage versioned datasets covering core flows, edge cases, negative scenarios, and safety validations
Validate conversational behavior (intent recognition, slot extraction, state transitions, business rules) and detect regressions

Must-have skills

4+ years in QA, data, ML, or AI-adjacent roles with production-system exposure
Hands-on testing of AI/LLM/NLU/conversational systems; handle non-deterministic behavior
Ability to design deterministic validations; experience with structured outputs (JSON schemas, tool usage, function calling)
EU work permit or CPLP nationality

Nice-to-haves

BSc/MSc in CS/Engineering/IT or related
Experience testing voice-based, conversational, or multimodal AI
Background in data quality, analytics workflows, or ML pipelines
Familiarity with AI observability/evaluation tooling; Python scripting
Location & work type: Location not specified; EU work permit or CPLP nationality required; no remote/work模式 stated.

Full Description

A Company you will be happy to grow with!;

At InnoTech, it´s all about connecting people with business through technology;

We care about you, your happiness, and your well-being;

We seek people who want to learn and further improve their skills with us – growth is part of our DNA, and that’s why we’re proud to be certified as a Great Place to Work and to be among the Top 50 Best Workplaces in Portugal!;

For us, sustainability is not just a word;

Integrity and social responsibility lead our behavior;

We dare to be different, ambitious, dream big, and deliver value like no other!;

You will have the opportunity to work with top-level partners in the banking area, insurance, retail, telecommunications, financial services, and so much more!;

We are 180+ InnoTechers, and we want your talent as an AI QA Specialist.

Your Mission Will Be

Define, evolve, and maintain evaluation strategies for AI, LLM, and conversational systems; Design and manage versioned datasets covering core flows, edge cases, negative scenarios, and safety validations; Validate conversational behavior including intent recognition, slot extraction, state transitions, and business rules; Detect regressions and evaluation drift across model, data, or prompt changes using reliable quality signals; Collaborate with QA and ML teams to embed AI evaluation into CI/CD pipelines and release processes.

You Will Need

4+ years of experience in QA, data, ML, or AI-adjacent roles with exposure to production systems; Hands-on experience testing AI, LLM, NLU, or conversational systems and handling non-deterministic behavior; Strong analytical skills with the ability to design deterministic validations where possible; Experience working with structured outputs such as JSON schemas, tool usage, or function calling; CPLP nationality or an EU work permit is required to facilitate employment.

Would Be Awesome If You Have

BSc or MSc in Computer Science/Engineering, Information Technology, or related fields; Experience testing voice-based, conversational, or multimodal AI systems; Background in data quality, analytics workflows, or ML pipelines; Familiarity with observability, monitoring, or evaluation tooling for AI systems; Scripting experience, preferably using Python.

With Us You Can Be a

TechThinker – people who want to re-imagine technology with us; TechSolver – strategic, resilient, and a little bit of a puzzle lover; InnoMinder – inhale innovation, exhale digital transformation; InnoTecher – no words needed. Just happiness, talent, and all the above!.

Are you convinced? Come see it for yourself.

Join our Top-Notch Team!

Show more Show less