Role & seniority: AI Agent Testing Specialist (mid-level; 2+ years in QA/testing)

Stack/tools: IVR/CCaaS platforms (Amazon Connect, Twilio, 3CX, etc.); conversational AI systems (ChatGPT, Dialogflow, Lex); API testing tools (Postman); basic scripting (Python)

Top 3 responsibilities

Design and execute scenario-based tests to evaluate AI agent behavior, accuracy, and naturalness in simulated call environments
Test call flows, intent recognition, response accuracy, and escalation handling; analyze logs and decision paths
Collaborate with developers, AI trainers, and QA teams to refine conversational logic and validate AI responses; contribute to test automation and continuous test plan improvements

Must-have skills

2+ years in software/QA testing, preferably with AI, IVR, or CCaaS
Familiarity with conversational AI systems (e.g., ChatGPT, Dialogflow, Lex)
Experience with test planning, scenario testing, and quality documentation; strong analytical and communication skills; user-perspective evaluation

Nice-to-haves

AI prompt design or conversational flow-building experience
Background in speech recognition, NLP, or voice application testing
Exposure to Python, Postman, or API testing tools
Location & work type: Not specified in the provided text; no explicit location or remote/on-site stated. Training and processes are provided; role appears collaborative and cross-functional.

Full Description

About the Role

We’re looking for a skilled and detail-oriented AI Agent Testing Specialist to help ensure the quality and performance of our AI-driven voice and conversational agents. This role goes beyond traditional software testing — you’ll be evaluating how AI behaves, responds, and performs in real-world call scenarios across our communication platforms.
If you enjoy testing interactive systems, analyzing conversational flows, and helping shape how intelligent virtual agents interact with customers, this role is for you.

Key Responsibilities

Design and execute scenario-based test cases to evaluate AI agent behavior, accuracy, and naturalness in simulated call environments.
Test AI call flows, intent recognition, response accuracy, and escalation handling.
Identify and document behavioral issues, misclassifications, or poor user experiences.
Collaborate with developers, AI trainers, and QA teams to refine conversational logic and performance.
Analyze call logs, transcripts, and AI decision paths to suggest improvements.
Support the fine-tuning and validation of AI responses based on training data and feedback.
Contribute to continuous test automation and test plan improvements.

Required Skills & Experience

2+ years of experience in software or QA testing, preferably with AI, IVR, or contact center solutions (CCaaS, Amazon Connect, Twilio, 3CX, etc.)
Familiarity with conversational AI systems (ChatGPT, Dialogflow, Lex, or similar).
Understanding of test planning, scenario testing, and quality documentation.
Strong analytical and communication skills.
Ability to think from the end-user perspective and evaluate how interactions “feel.”
Comfortable working in collaborative, cross-functional environments.

Preferred Experience

Experience with AI prompt design or conversational flow building.
Background in speech recognition, NLP, or voice application testing.
Exposure to Python, Postman, or API testing tools.

Training & Support

We provide hands-on training on our specific AI testing platform, tools, and process.
Candidates with a strong QA foundation and a willingness to learn are highly encouraged to apply.

What we offer

Market-competitive salary
Career growth opportunities
Learning & development programs
On-the-job training
Medical Coverage (IPD)

AI Conversational Testing Specialist