Role & seniority: GenAI Tester (mid-senior level, 5+ years IT/software development)

Stack/tools: Azure Cloud with Azure AI Services, Azure OpenAI, AI Search; Python; LangChain, LangGraph; Cosmos DB; Azure Storage (Blob/Queue/Table); Flask; Azure Functions; HTTP; APIM; RAG evaluation (grounding, faithfulness, precision/recall); pytest; observability (Application Insights)

Top 3 responsibilities

Test RAG bots, Agentic AI bots, and Gen AI applications on Azure, with secure configuration and scalable data layers
Validate prompts, tuning, guardrails, and mitigate hallucinations across system/retrieval/tool-use prompts; ensure end-to-end reliability
Establish automated testing (pytest) and RAG evaluation; ensure observability and end-to-end traceability; tune SLAs, scaling, and cost/performance

Must-have skills

3+ years hands-on Azure Cloud experience with Azure AI Services, AI Search, Azure OpenAI, Cosmos DB, and Azure Storage
Proven prompt engineering and tuning skills; expertise in test automation with pytest and RAG/factoring grounding techniques
Experience validating APIs with automated testing tools and evaluating AI outputs (grounding, semantic ranking, context filtering)

Nice-to-have skills

LangGraph experience for agentic flows
Familiarity with CI/CD pipelines (Azure DevOps, GitHub Actions)
Location & work type: Hybrid role; location not specified; remote/hybrid acceptable per description

Full Description

GenAI Tester

Role Overview

We are seeking a Gen AI Tester who can test and validate enterprise grade RAG, Agentic AI and Generative AI applications on Azure This hybrid role combines hands on designing unit test cases along with rigorous testing and evaluation to ensure solutions are reliable grounded and accurate The ideal candidate will have strong technical expertise in Azure AI services Python frameworks and automated testing methodologies Candidate should also have hands on experience in evaluating matrices like Accuracy Correctness Faithfulness and other similar matrix in RAG based systems applications

Experience Requirements

5+ years of overall experience in core IT and software development 3 years of handson experience with Azure Cloud with Azure AI Services 2 years of experience in Generative AI RAG LLMs Azure Open AI Python Lang Chain Lang Graph and related technologies in testing AI Agents

Key Responsibilities

Test RAG bots Agentic AI bots and Gen AI applications using Azure Open AI Azure AI Search Python Flask Azure Functions HTTP Queue Blob Timer and APIM with secure configuration via environment variables Test scalable data layers using Cosmos DB and Azure Storage Blob Queue Table optimize chunking embeddings and hybrid vector search pipelines Validate prompts and tuning across system prompts retrieval prompts and tooluse prompts test guardrails to minimize hallucinations Establish automated testing with pytest and RAG evaluation using Ragas faithfulness precision recall answer correctness Ensure observability with Application Insights advanced logging and endtoend traceability Test orchestrate workflows using LangChainLangGraph for stateful agent flows Test production SLAs by tuning rate limits scaling AI Search tiers and optimizing cost performance

Must Have Skills

3 years of hands on experience with Azure Cloud with Azure AI Services, AI Search Azure Open AI Cosmos DB and Azure Storage Proven experience with prompt engineering and tuning techniques Expertise in test automation using pytest and evaluation with Ragas knowledge of grounding techniques hybrid search semantic ranking and context filtering Hands on experience in using any testing tool to automate and validate APIs

Nice To Have Skills

Experience with Lang Graph for agentic flows Familiarity with CICD pipelines Azure DevOps GitHub Actions

Preferred Certifications

Microsoft Certified Azure AI Engineer AI102

This job is provided by Shine.com