**Role & seniority: ** Senior AI Quality / Reliability Engineering; 5+ years relevant experience

**Stack/tools: ** Python; automated testing frameworks; API/UI/integration/performance/end-to-end testing; CI/CD quality gates; (nice-to-haves: Playwright, Selenium; LLM evaluation frameworks)

**Top 3 responsibilities: **

Build automated evaluation frameworks for agents/RAG/prompts/tools/workflows/model outputs (benchmarks, metrics, thresholds)
Create test assets: benchmark datasets, golden responses, regression suites; define measurable quality criteria
Validate AI safety/quality via hallucination, grounding, citation accuracy, prompt injection, tool selection, and multi-agent coordination; include production monitoring/drift
Must-have skills:
- Bachelor’s in CS/SE/IS or related field
- 5+ years in quality engineering, test automation, or reliability engineering
- Strong Python for reusable automated test frameworks
- Experience with API/UI/integration/regression/CI-CD testing and measurable quality gates
- Ability to translate requirements and AI behaviors into quantifiable evaluation criteria
- Experience testing generative AI/RAG/copilots/AI agents
Nice-to-haves:
- Playwright/Selenium; LLM evaluation and prompt testing
- Agent-powered test generation, workflow validation, intelligent UAT
- Adversarial testing / Responsible AI testing; production quality monitoring
Location & work type: Not specified

Full Description

Project Description

Own AI quality, automated testing, reliability engineering, and agent-powered QA automation for the platoon.

Responsibilities

Build automated evaluation frameworks for agents, RAG, prompts, tools, workflows, and model outputs.
Create benchmark datasets, golden responses, regression suites, evaluation metrics, and release thresholds.
Test hallucination, grounding, citation accuracy, prompt injection, tool selection, and multi-agent coordination.
Develop agents that generate test cases, validate API and workflow outputs, and support intelligent UAT.
Build UI, API, integration, performance, end-to-end automation, and CI/CD quality gates.
Monitor production quality, drift, reliability, and recurring failure patterns.

Mandatory Skills Description

Bachelor's degree in Computer Science, Software Engineering, Information Systems, or a related field.
5+ years of software quality engineering, test automation, or reliability engineering experience.
Strong Python skills and experience building reusable automated test frameworks.
Experience with API, UI, integration, regression, and CI/CD-based testing, including measurable quality gates.
Ability to translate requirements, business workflows, and AI behaviors into measurable quality criteria.
Experience testing generative AI, RAG, copilots, or AI agents.

Nice-to-Have Skills Description

Experience with Playwright, Selenium, LLM evaluation frameworks, and prompt testing.
Experience developing agent-powered test generation, workflow validation, or intelligent UAT assistants.
Experience with adversarial testing, Responsible AI testing, and production quality monitoring.

Languages

English: B2 Upper Intermediate