Full Description

Key Responsibilities

• Design and execute test plans for voice and chat AI agents.
• Perform load testing, latency measurement, and barge-in handling for voicebots.
• Test chatbot flows across different LLMs, embeddings, tools, and agents (LangChain/LangGraph).
• Build and maintain automated testing frameworks for regression testing (e.g., prompt consistency, tool behavior).
• Evaluate intent accuracy, NLU robustness, and fallback handling.

Simulate edge cases: noisy environments, overlapping speech, disconnections, multilingual input. Report bugs, behavioral inconsistencies, hallucinations, and voice UX issues. Monitor and interpret analytics, call logs, response latencies, prompt token usage, etc. Collaborate closely with engineers and product leads to prioritize fixes.

Requirements

Skills and Qualifications

• 3+ years of QA/testing experience with a focus on conversational AI systems.
• Familiarity with LangChain, LangGraph, or prompt-driven workflows (or interest in learning quickly).

Experience with voice automation tools: Twilio, Dialogflow, Amazon Lex, or Synthflow. Strong knowledge of testing tools (Postman, Playwright, pytest, JMeter, etc.).

Understanding of NLP testing: intent/slot recognition, confusion matrix, false positives/negatives. Excellent debugging, test documentation, and bug reporting skills.

Nice to Have Experience testing systems with real-time voice TTS/STT, barge-in, and silence detection.

Familiarity with LLM-based applications: OpenAI, Claude, Gemini, etc.

Hands-on with QA automation: CI/CD test suites, conversational analytics, or prompt eval tools like LangSmith, Promptfoo, or TruLens. Knowledge of automotive, e-commerce, or support systems. Arabic language testing experience. Do you want to be part of this story? Apply now!