**Role & seniority: ** Senior AI QA / Evaluation Engineer (senior)

**Location & work type: ** Baku, Azerbaijan — Hybrid / On-site

**Stack/tools: **

Evaluation frameworks: Ragas, DeepEval, promptfoo (or equivalent)
Metrics: groundedness, faithfulness, context precision/recall, answer correctness, tool-call accuracy
Testing/security: OWASP Top 10 for LLMs, garak, PyRIT
Automation/CI: Python, pytest, API testing, Playwright, CI-based gates
Observability: Langfuse, Arize Phoenix (or equivalent)
Methods: repeated sampling, bootstrapped confidence intervals
Defect management: formal defect/issue tracking; audit evidence pack
Top 3 responsibilities:
- Build versioned golden datasets and evaluation harnesses for LLM/ML/agents
- Act as quality gate for production releases by enforcing evaluation baselines in CI
- Run adversarial/security tests (prompt injection, data leakage, unsafe tool use) and manage incident triage + audit evidence
Must-have skills:
- 4+ years in QA/test engineering with test automation ownership
- Hands-on experience evaluating LLM/ML outputs
- Strong Python for evaluation tooling; pytest/API testing experience
- Formal defect management experience
- English fluency; based in Baku or willing to relocate
Nice-to-haves:
- Audit evidence experience in regulated sectors
- Experience with model risk management framewo

Full Description

About DataSpecta

DataSpecta is a Data & AI consulting and engineering company helping organizations transform data into measurable business value through advanced analytics, modern data platforms and enterprise AI solutions.

We design and deliver solutions across both on-premise and cloud environments, working closely with enterprise customers from discovery through production.

Alongside customer projects, we build our own AI products such as CogniSpecta (enterprise knowledge & cognitive search) and CodeSpecta (AI-powered software engineering).

We're looking for a Senior AI QA / Evaluation Engineer based in Baku, Azerbaijan, with strong test automation expertise and hands-on experience evaluating LLM and ML systems.

You will own quality assurance for production AI systems in a regulated enterprise environment, acting as the quality gate for every production release.

Location & Work Model

Location: Baku, Azerbaijan

Work model: [Hybrid / On-site]

What You Will Work On Evaluation harnesses and golden datasets for enterprise AI agents Evaluation of RAG assistants and report generation agents Adversarial and security testing for LLM applications Audit evidence and quality governance

Responsibilities Build versioned golden datasets and evaluation harnesses Gate production releases on evaluation baselines in CI Revalidate agents at each model version change Run quality control on recurring reports Run adversarial tests for prompt injection, data leakage and unsafe tool use Triage support and agent output incidents Maintain the audit evidence pack

Required Qualifications Based in Baku or willing to relocate 4+ years in QA or test engineering, including test automation ownership Proven experience evaluating LLM or ML outputs Strong Python skills for evaluation tooling Experience with formal defect management Fluent English

Technical Skills

Evaluation frameworks: Ragas, DeepEval, promptfoo or equivalent

Metrics: Groundedness, faithfulness, context precision/recall, answer correctness, tool call accuracy

Statistics: Repeated sampling, bootstrapped confidence intervals

Automation: Python, pytest, API testing, Playwright, CI

Adversarial testing: OWASP Top 10 for LLMs, garak, PyRIT

Observability: Langfuse, Arize Phoenix or equivalent

Multilingual: Azerbaijani and mixed-language evaluation

Preferred Qualifications (a plus) Audit evidence experience in a regulated sector Model risk management frameworks

What We Value Curiosity about solving business problems with AI Comfort across business, data and technology End-to-end ownership Continuous learning

LinkedIn: https://www.linkedin.com/company/innovance-consultancy

LinkedIn: https://www.linkedin.com/company/dataspecta

Instagram: https://www.instagram.com/innovanceconsultancy

Instagram: https://www.instagram.com/dataspecta

Senior AI QA / Evaluation Engineer

Full Description

Follow us