
Senior QA Engineer (AI Testing)
Gallagher • Colombo, Western Province, Sri Lanka
**Role & seniority: ** Senior QA Engineer (AI Testing) — senior individual contributor; reports to QA Practice Lead
**Location & work type: ** Colombo, Sri Lanka — Hybrid
**Stack/tools: **
-
AI testing: generative AI / ML, intelligent automation, retrieval-augmented generation (RAG), prompt engineering
-
CI/CD: Azure DevOps (CI platforms)
-
Testing tools: Postman, JMeter
-
Methods: Agile; secure software engineering practices
-
Top 3 responsibilities:
-
Develop and maintain end-to-end test strategies for AI solutions (testing, evaluation, production monitoring)
-
Evaluate AI behavior for accuracy, robustness, fairness, and safety; test AI risks (hallucinations, data leakage, model drift)
-
Build automated regression suites and integrate into CI/CD; collaborate on acceptance criteria and quality-risk communication
-
-
Must-have skills:
-
Proven software testing experience with AI-specific testing
-
Strong understanding of AI risks and testing approaches
-
Ability to integrate automated test suites into CI pipelines
-
Knowledge of performance, accessibility, and security testing
-
-
Nice-to-haves:
-
ISTQB certification/training in AI or ML
-
Experience applying advanced testing tools/techniques and staying current with industry trends
-
Full Description
Introduction
At Gallagher, we help clients face risk with confidence because we believe that when businesses are protected, they’re free to grow, lead, and innovate. You’ll be backed by our digital ecosystem: a client-centric suite of consulting tools making it easier for you to meet your clients where they want to be met. Advanced data and analytics providing a comprehensive overview of the risk landscape is at your fingertips. Here, you’re not just improving clients' risk profiles, you’re building trust. You’ll find a culture grounded in teamwork, guided by integrity, and fueled by a shared commitment to do the right thing. We value curiosity, celebrate new ideas, and empower you to take ownership of your career while making a meaningful impact for the businesses we serve. If you’re ready to bring your unique perspective to a place where your work truly matters; think of Gallagher.
Overview
The Senior QA Engineer (AI Testing) ensures the quality and reliability of AI-enabled solutions across their lifecycle, including testing, evaluation, and production monitoring. The role involves collaboration with stakeholders to define acceptance criteria, assess risks, and implement testing strategies for AI solutions such as generative AI, machine learning, and intelligent automation. Based in Colombo, Sri Lanka, this position reports to the QA Practice Lead.
How you'll make an impact
Develop and maintain comprehensive test strategies for AI-enabled solutions. Evaluate AI behaviour for accuracy, robustness, fairness, and safety. Test AI-specific risks, including hallucination, data leakage, and model drift. Build automated regression suites and integrate them into CI/CD pipelines. Collaborate with stakeholders to clarify requirements and communicate quality risks. Monitor AI quality in production and contribute to continuous improvement. Stay updated on industry trends and introduce advanced testing tools and techniques.
About you
Proven experience in software testing, including AI-specific testing. Expertise in AI risks, retrieval-augmented generation, and prompt engineering. Familiarity with secure software engineering practices and Agile methodologies. Experience with performance, accessibility, and security testing. Knowledge of CI platforms (e.g., Azure DevOps) and tools like Postman and JMeter. ISTQB certification and training in AI or machine learning are advantageous. This role is ideal for candidates passionate about advancing AI testing practices in a collaborative environment. #LI-Hybrid