
Quality Assurance Engineer
Mphasis • Mumbai City, Maharashtra, India
-
Role & seniority
- AI/ML Test Engineer (QA/SDET) — 7+ years total experience; 1–2 years in AI/ML domain (also described as 6+ years QA/SDET with AI/ML focus)
-
Stack/tools
-
Testing/automation: Playwright, Selenium (listed), PyTest, API testing via Postman, human-led verification
-
Programming: Python
-
Back-end/data: MongoDB, PostgreSQL, key-value caching, vector DBs Chroma DB / PGVector
-
AI/agent tech: LLMs, prompts/tokens/context/hallucinations, agent workflows, tool/function calling, RAG, memory, multi-agent
-
Cloud/infra: Azure/AWS/GCP (AI services), Git, CI/CD, Docker
-
Agent orchestration: Agents SDK / orchestration frameworks (e.g., tool/function calling reliability)
-
-
Top 3 responsibilities
-
Evaluate agent decision-making & safety: verify scope decisions, planning steps, tool-call sequences; test guardrails/autonomy levels
-
Audit and supervise agentic testing systems: review AI-generated tests for correctness and coverage; tune what the system chooses to test
-
Verify orchestration reliability & end-to-end workflows: validate agent/LLM pipelines (single/multi-agent, RAG, memory) and troubleshoot via execution logs
-
-
Must-have skills
-
Strong QA/SDET fundamentals; manual + API testing experience
-
Strong Playwright proficiency
-
Clear understanding of Agentic AI: planning/execution, tool/function calling, memory, non-deterministic outputs, orchestratio
-
Full Description
AI/ML Test Engineer – Job Description
Minimum 7 years of experience Job Description –
Evaluate agent decision‑making: Assess whether an agent’s scope decisions, planning steps, and tool‑calling sequences are correct and safe Strong Playwright experience and understanding Agentic AI experience and understanding API
Audit AI‑generated tests: Review assertions and test cases produced by agentic testing systems to ensure they verify the right behaviours.
Supervise agentic testing systems: Tune and correct what the AI testing system chooses to cover, rather than writing every test manually
Verify orchestration reliability: Validate the orchestration framework (e.g., Agents SDK) and ensure tool/function‑calling reliability.
Check guardrails and autonomy levels: Test how much autonomy agents have in production and whether guardrails prevent unsafe actions
Build independent verification layers: Maintain human‑owned checks that validate AI‑generated outputs. 6+ Years of Total IT experience in QA/SDET and Testing, with 1to 2 years in the field of AI/ML domain. Test Agentic AI applications and AI-Driven Workflows Validate agent workflows complete flow from start to end Test LLM-based applications for accuracy, relevance, consistency, and hallucinations. Test multi-agent workflows and communication between agents. Test RAG (Retrieval-Augmented Generation) workflows and knowledge retrieval. Validate agent memory and context handling. Perform API testing using Postman or similar tools. Perform database testing and backend data validation using MongoDB/PostgreSQL. Knowledge on Key Value Caching, Vector Databases like Chroma DB/PGVector Perform basic prompt injection and AI security/guardrail testing. Analyse agent execution logs and identify the root cause of failures. Create and maintain test cases, test scenarios, bug reports, and QA documentation. Work closely with AI/ML and development teams to improve overall agent reliability. Required Skills Strong knowledge of Software Testing / QA fundamentals. Experience with Manual Testing and API Testing. Understanding of Generative AI, LLMs, prompts, tokens, context, and hallucinations. Understanding of Agentic AI concepts and agent workflows. Knowledge of AI agents, tool/function calling, planning, execution, and memory. Basic understanding of RAG and vector databases. Ability to design test cases for non-deterministic AI outputs. Strong analytical, debugging, and problem-solving skills. Perform database testing and backend data validation using MongoDB/PostgreSQL. Knowledge on Key Value Caching, Vector Databases like Chroma DB/PGVector. Python knowledge and test automation experience. Selenium, Playwright, PyTest, or similar automation tools. Knowledge of AI evaluation techniques such as groundedness, relevance, faithfulness, and response quality. Experience with Azure/AWS/GCP and AI services. Git, CI/CD, and Docker knowledge. Experience testing multi-agent or AI orchestration systems. Strong QA/SDET engineering background Understanding of agentic orchestration frameworks Ability to evaluate AI generated code, tests, and decisions Risk based thinking and judgment-heavy evaluation