Full Description

Role Overview We are seeking an experienced AI Test Lead to lead end-to-end testing and quality assurance of Generative AI applications. This role bridges AI quality engineering, test automation, and customer engagement. You will collaborate with QA Managers, AI/ML Engineers, Prompt Engineers, Full Stack Developers, and DevOps Engineers to validate production-ready, scalable AI applications against industry AI testing standards, while serving as the primary testing point of contact for customers. Key Responsibilities AI Testing Strategy & Delivery Management

Lead full lifecycle of AI testing activities: test strategy, planning, design, execution, defect management, and sign-off for GenAI applications Define and implement test strategies for LLM-powered applications including RAG systems, conversational AI, and AI agents

Plan and manage functional, non-functional, and AI-specific testing: model evaluation, prompt regression, bias/fairness, robustness, hallucination detection, and explainability validation Establish test estimation, sprint-level test planning, milestones, and delivery timelines aligned with agile development cycles Track and report test coverage, quality metrics, and release readiness to QA leadership and stakeholders Customer Engagement & Test Management Act as the primary testing point of contact for customers; lead requirement discussions, status reviews, and defect triage calls Translate customer business requirements and acceptance criteria into comprehensive AI test plans and test scenarios

Manage day-to-day testing activities of the QA team: task allocation, progress monitoring, risk identification, and escalation management Present test results, quality dashboards, and risk assessments to customer stakeholders in business-focused language Drive UAT support, release sign-off, and post-release quality monitoring in coordination with customer teams AI Testing Standards & Quality Governance

Implement AI testing practices aligned with industry standards: ISO/IEC 42001 (AI management systems) Apply responsible AI and risk frameworks such as NIST AI RMF and EU AI Act requirements to test planning and evidence collection Define quality gates, entry/exit criteria, and audit-ready test documentation for AI systems

Establish evaluation benchmarks and metrics for LLM outputs: accuracy, relevance, groundedness, toxicity, and consistency Test Automation & Tooling Design and build test automation frameworks primarily using Python (Pytest, Selenium, Playwright, Requests) Automate AI/LLM evaluation pipelines using frameworks such as DeepEval, Ragas, LangSmith, or promptfoo Integrate automated test suites into CI/CD pipelines (GitHub Actions, Jenkins) with reporting and alerting Drive API, integration, and performance testing for AI services and microservices Mentor QA engineers on Python scripting, automation best practices, and AI testing techniques Required Skills & Qualifications Experience 8+ years of overall experience in software testing and quality assurance 1–2 years of hands-on experience in AI/ML and GenAI testing activities Proven experience interacting with customers and managing end-to-end testing activities and QA teams Track record of testing production-grade AI applications through complete release cycles Technical Knowledge Strong understanding of LLMs (GPT, Claude, Llama), prompt engineering, RAG architectures, and AI agents from a testing perspective

Hands-on expertise in AI-specific testing: model evaluation, bias/fairness testing, adversarial/robustness testing, hallucination detection, and prompt regression testing Strong automation skills with Python scripting; experience with Pytest, Selenium/Playwright, and API testing tools (Postman, Requests)

Familiarity with LLM evaluation tools and frameworks: DeepEval, Ragas, LangSmith, Hugging Face evaluate, or equivalent Working knowledge of cloud platforms (AWS, Azure, GCP), CI/CD pipelines, and test management tools (Jira, Zephyr, TestRail) AI Testing Standards

Knowledge of AI testing and quality standards: ISO/IEC 42001, and ISTQB CT-AI syllabus

Understanding of responsible AI frameworks and regulations: NIST AI RMF, EU AI Act, and data privacy considerations in AI testing Education Bachelor's or Master's degree in Computer Science, Engineering, AI/ML, Data Science, or related field ISTQB Certified Tester – AI Testing (CT-AI) or equivalent AI/QA certifications are a plus What This Role Offers Lead cutting-edge GenAI testing programs with high visibility to customers and senior leadership Ownership of AI quality strategy from test planning through production sign-off Work with talented AI architects, engineers, and domain experts Continuous learning and growth in the rapidly evolving AI testing and quality engineering space