Full Description

Key Responsibilities

Lead QA strategy and define testing approaches for agentic AI systems and ML applications Design and implement comprehensive test automation frameworks using Python Develop specialized testing methodologies for AI models including accuracy, fairness, robustness, and drift detection Build and maintain CI/CD pipelines with integrated automated testing Test complex Azure architectures including microservices, serverless functions, and distributed systems Validate data pipelines, ETL processes, and analytics workflows in Azure Databricks Perform advanced performance testing, load testing, and scalability validation Establish quality metrics, KPIs, and reporting dashboards for AI applications Mentor junior QA engineers and promote best practices across the team Collaborate with architects and developers on testability and quality requirements Implement monitoring and validation strategies for production AI systems Lead root cause analysis for critical production issues Evaluate and integrate new testing tools and frameworks Ensure compliance with security, privacy, and regulatory requirements

QA Leadership

Required Skills & Qualifications

8+ years of experience in software QA with at least 2+ years testing AI/ML systems Proven track record of establishing QA processes and testing frameworks Experience leading testing efforts for production-grade applications Strong understanding of QA best practices, methodologies, and industry standards

Use LLM or AI for testing or Python working experience in testing.

Technical Expertise

Expert-level proficiency in Python for test automation and scripting Advanced SQL skills for complex data validation and quality checks Deep knowledge of Azure services (Functions, ML, OpenAI, Databricks, Storage, API Management) Extensive hands-on experience with Azure Databricks including testing data workflows Strong experience building and maintaining CI/CD pipelines (Azure DevOps, GitHub Actions) Proficiency with test automation frameworks (pytest etc...) Experience with API testing and microservices validation

AI/ML Testing Expertise

Strong understanding of Machine Learning, Natural Language Processing, and Deep Learning Specialized knowledge of AI/ML testing strategies (model evaluation, data quality, bias detection, adversarial testing) Experience testing LLM-based applications including prompt validation and response quality Knowledge of MLOps practices and testing ML pipelines Understanding of A/B testing and experimentation frameworks Familiarity with model monitoring and drift detection

Azure & Databricks

Deep knowledge of Azure architecture and deployment patterns Experience testing serverless applications and event-driven systems Proficiency with Azure monitoring tools (Application Insights, Log Analytics) Strong understanding of Databricks notebooks, jobs, and cluster configurations Experience with infrastructure testing and validation