AI Quality & Reliability Engineer – QA/SRE
D4 Insight • Abu Dhabi, Abu Dhabi Emirate, United Arab Emirates
**Role & seniority: ** AI Quality & Reliability Engineer (7+ years). Hybrid role combining AI QA, model evaluation, and SRE/production readiness.
**Location & work type: ** Abu Dhabi, UAE — hybrid.
**Stack/tools: **
-
Languages/automation: Python, Bash
-
Testing/QA: PyTest, Selenium, Robot Framework, CI/CD test suites (GitHub Actions)
-
Cloud/SRE: Azure (Azure Monitor), AWS (CloudWatch), networking/scaling/serverless reliability
-
IaC/ops: Terraform
-
Load/chaos: K6 or JMeter, Chaos Engineering
-
Monitoring/metrics: Grafana, Prometheus, SLIs/SLOs
-
AI evaluation: DeepEval, Ragas, LangSmith; metrics like ROUGE/BLEU; RAG evaluation
-
Top 3 responsibilities:
-
Build and run automated AI testing frameworks for Azure OpenAI / AWS Bedrock (hallucinations, accuracy, content safety, latency).
-
Ensure reliability/performance via load/chaos testing, recovery automation, and SLIs/SLO monitoring/alerting.
-
Support security and compliance automation for banking requirements (data residency/privacy) and ongoing scan/compliance checks.
-
-
Must-have skills:
-
Strong Python automation/testing (PyTest, Selenium, Robot Framework).
-
Hands-on Azure or AWS reliability (networking, scaling, serverless) and monitoring (Azure Monitor/AWS CloudWatch or similar).
-
Practical AI output evaluation (prompting/RAG evaluation; benchmarking/quality metrics).
-
Experience defining and monitoring SLIs/SLOs.
-
-
Full Description
Location: Abu Dhabi, UAE
Experience: 7+ Years
Role Overview
We are seeking for AI Quality & Reliability Engineer to own the production readiness of cloud-based AI solutions.
This hybrid role combines automated software testing, AI model evaluation, and Site Reliability Engineering (SRE). The role focuses on building automated frameworks to validate AI outputs while ensuring Azure/AWS infrastructure is resilient, performant, and aligned with banking standards.
Key Responsibilities
AI Quality & Automated Testing
Design and execute automated testing frameworks for AI services such as Azure OpenAI and AWS Bedrock.
Test AI solutions for
- Model hallucinations
- Accuracy
- Content safety
- Latency
- Build CI/CD-integrated test suites using Python to validate application logic and infrastructure state.
- Reliability & Performance
Implement Chaos Engineering and load testing for banking-scale web and mobile backends. Develop automated recovery scripts to support high availability. Define and monitor SLIs and SLOs. Configure advanced monitoring and alerting using Azure Monitor or AWS CloudWatch.
Security & Compliance
Automate security scans and compliance checks. Ensure AI data handling aligns with banking data residency and privacy requirements. Identify inefficient cloud resource usage and expensive failing tests during testing.
Required Skills & Experience
7+ years of relevant experience.
High proficiency in Python and automation frameworks such as
- PyTest
- Selenium
- Robot Framework
- Strong hands-on experience with Azure or AWS, particularly networking, scaling, and serverless reliability.
Understanding of Prompt Engineering and AI output evaluation, including
- RAG evaluation
- ROUGE/BLEU
- Custom LLM benchmarks
- Experience with Grafana, Prometheus, Azure Monitor, AWS CloudWatch, or similar monitoring tools.
- Experience with Bash, GitHub Actions, Terraform, K6/JMeter.
- Exposure to AI evaluation tools such as DeepEval, Ragas, or LangSmith.
📩 Apply Now: careers@d4insight.com
