Gen Ai Tester
Tekshiras Software Services Private Limited • Hyderabad, Telangana, India
Role & seniority: Senior QA Engineer / Test Architect (AI agent testing, End-to-end QA for enterprise SaaS integrations)
Stack/tools: Python, pytest (async support, fixtures, parametrization, mocking, pytest-asyncio); FastAPI; LangGraph/LangSmith; LLM evaluation patterns; semantic similarity scoring; SQL; Workday and ServiceNow REST APIs; OAuth/API keys; Langfuse/observability; CI/CD (GitHub Actions); Docker Compose; Allure/pytest-html
Top 3 responsibilities
-
Design and implement end-to-end test frameworks for AI agent applications integrating Workday and ServiceNow; develop specialized tests for LLM output, SQL generation, and enterprise SaaS flows
-
Build automated evaluation pipelines (automated evals, semantic similarity scoring; LLM-as-judge validation) and maintain continuous monitoring of AI agent performance
-
Conduct regression testing, API/integration testing, data validation (Workday payroll data, ServiceNow tickets), and ensure cross-system data reconciliation and QA coverage across environments
Must-have skills
-
Python testing with pytest (async, fixtures, parametrization, mocking/patching, pytest-asyncio)
-
AI agent testing (LangSmith integration, automated evals, LLM-as-judge, prompt validation, LangGraph multi-node workflows, semantic similarity scoring)
-
API integration testing (FastAPI endpoints, REST contracts, JSON schema, OAuth/API keys, ServiceNow REST API)
-
Test data management (SQL, data migration, JSON data generation, dat
Full Description
Design and execute end-to-end tests for an AI agent application that assists users with common HR tasks via Workday and ServiceNow integration. Build automated test frameworks using pytest for Python backend (FastAPI/LangGraph agents) and specialized testing approaches for LLM output validation, SQL query generation, and enterprise SaaS integrations. Implement automated evaluation pipelines with automated evals and semantic similarity scoring. Ensure quality through evals, regression testing, LLM-as-judge validation, and continuous monitoring of AI agent performance.
Top Required Skills
-
Python Testing Frameworks - pytest with async support, fixture management, parametrized tests, mocking/patching, pytest-asyncio for LangGraph workflows
-
AI Agent Testing - Lang Smith integration, automated evals, LLM-as-judge evaluation patterns, prompt validation, LangGraph multi-node workflow testing, semantic similarity scoring
-
API Integration Testing - FastAPI endpoint validation, REST API contract testing, JSON schema validation, authentication flows (OAuth, API keys), Langfuse/observability platform integration, ServiceNow REST API testing
-
Test Data Management - SQL, data migration, JSON test data generation, data normalization strategies, production trace capture/sanitization
-
Workday & ServiceNow Integration Testing - Workday payroll data validation (earnings, deductions, PTO), ServiceNow ticket creation/updates, REST API/RaaS testing, cross-platform data reconciliation, multi-system workflow validation
-
Performance & CI/CD - Load testing, LLM latency benchmarking, GitHub Actions workflows, Docker Compose test environments, Allure/pytest-html reporting, automated eval pipelines
