Role & seniority: GenAI Tester, mid to senior level (5+ years IT/software, 3+ years Azure experience)

Stack/tools: Azure AI Services, Azure OpenAI, AI Search, Cosmos DB, Azure Storage, Python, Flask, Azure Functions, HTTP/Queue/Blob/Timer, APIM, LangChain, LangGraph, RAG, grounding techniques, experiential focus on prompts and tuning

Top 3 responsibilities

Test RAG bots, Agentic AI bots, and GenAI apps using Azure/OpenAI stack and secure configurations; validate performance across services and pipelines
Validate prompts, tuning, system/retrieval/tool-use prompts and guardrails to minimize hallucinations; assess accuracy, correctness, faithfulness
Establish automated testing with pytest and RAG evaluation (faithfulness, precision, recall, answer correctness); ensure observability and end-to-end traceability

Must-have skills

3+ years hands-on Azure Cloud experience with Azure AI Services, AI Search, Azure OpenAI; experience with Cosmos DB and Azure Storage
Proven prompt engineering and tuning skills; strong test automation experience (pytest) and RAG evaluation
Experience validating APIs with testing tooling and expertise in grounding techniques, hybrid search, semantic ranking, and context filtering

Nice-to-haves

Lang Graph experience for agentic flows
Familiarity with CI/CD pipelines (Azure DevOps, GitHub Actions)

Location & work type: Hybrid role; location not specified (flexible/hybrid arrangement)

Full Description

GenAI Tester

Role Overview

We are seeking a Gen AI Tester who can test and validate enterprise grade RAG, Agentic AI and Generative AI applications on Azure This hybrid role combines hands on designing unit test cases along with rigorous testing and evaluation to ensure solutions are reliable grounded and accurate The ideal candidate will have strong technical expertise in Azure AI services Python frameworks and automated testing methodologies Candidate should also have hands on experience in evaluating matrices like Accuracy Correctness Faithfulness and other similar matrix in RAG based systems applications

Experience Requirements

5+ years of overall experience in core IT and software development 3 years of handson experience with Azure Cloud with Azure AI Services 2 years of experience in Generative AI RAG LLMs Azure Open AI Python Lang Chain Lang Graph and related technologies in testing AI Agents

Key Responsibilities

Test RAG bots Agentic AI bots and Gen AI applications using Azure Open AI Azure AI Search Python Flask Azure Functions HTTP Queue Blob Timer and APIM with secure configuration via environment variables Test scalable data layers using Cosmos DB and Azure Storage Blob Queue Table optimize chunking embeddings and hybrid vector search pipelines Validate prompts and tuning across system prompts retrieval prompts and tooluse prompts test guardrails to minimize hallucinations Establish automated testing with pytest and RAG evaluation using Ragas faithfulness precision recall answer correctness Ensure observability with Application Insights advanced logging and endtoend traceability Test orchestrate workflows using LangChainLangGraph for stateful agent flows Test production SLAs by tuning rate limits scaling AI Search tiers and optimizing cost performance

Must Have Skills

3 years of hands on experience with Azure Cloud with Azure AI Services, AI Search Azure Open AI Cosmos DB and Azure Storage Proven experience with prompt engineering and tuning techniques Expertise in test automation using pytest and evaluation with Ragas knowledge of grounding techniques hybrid search semantic ranking and context filtering Hands on experience in using any testing tool to automate and validate APIs

Nice To Have Skills

Experience with Lang Graph for agentic flows Familiarity with CICD pipelines Azure DevOps GitHub Actions

Preferred Certifications

Microsoft Certified Azure AI Engineer AI102

This job is provided by Shine.com