Full Description

Role Description

Job Title

QE Architect – AI / LLM Systems

Role Summary

We are looking for a visionary QE Architect – AI/LLM Systems to architect, define, and drive the overall Quality Engineering (QE) strategy for next-generation AI products. This role will focus on building scalable quality architectures, AI evaluation frameworks, and automated testing pipelines to ensure reliable, safe, and high-quality AI-driven user experiences.

The ideal candidate will bring strong thought leadership and deep technical expertise, working at the intersection of AI/ML, LLM systems, software engineering, and quality governance.

Key Responsibilities

QE Architecture & Strategy Define and own the end-to-end quality architecture for all AI and LLM initiatives across the organization.

Design enterprise-level QE frameworks and reusable components for

• Conversational AI applications and chatbots
• Knowledge-management bots and RAG systems
• Semantic and vector-based text search
• Image search and multimodal AI systems
• Generative AI platforms
• Establish scalable testing pipelines for model evaluation, data validation, and automation.
• AI / LLM Evaluation Frameworks

Architect comprehensive evaluation systems for

• Prompt testing and scenario-based validation
• LLM output quality, safety, bias, and consistency
• Hallucination detection and mitigation
• RAG correctness, grounding accuracy, and knowledge integrity
• Search relevance and ranking metrics
• Build automated scorecards and continuous evaluation dashboards.
• Automation & Infrastructure

Design and implement automation frameworks for

• LLM APIs and chat agents
• Multimodal AI pipelines
• Vector databases and semantic search services

Architect model regression detection using

• Golden datasets
• Synthetic test data generation
• LLM-as-a-Judge approaches
• Self-evaluation and multi-agent evaluation techniques
• Integrate AI test harnesses into CI/CD and LLMOps pipelines.
• Data Quality & Test Data Strategy

Define enterprise-wide AI test data management strategies, including

• Ground-truth datasets
• Benchmark datasets
• Adversarial and edge-case inputs
• Safety and compliance-focused test scenarios
• Architecture Reviews & Cross-Team Leadership