Full Description

We are looking for an AI QA Automation Engineer (L5) to ensure the quality, reliability, and performance of AI-powered systems, including APIs, agentic workflows, and Retrieval-Augmented Generation (RAG) pipelines. This role goes beyond traditional QA automation. You will work at the intersection of quality engineering and AI systems, building automated validation frameworks for LLM-driven behavior and partnering closely with ML and full-stack teams to ship reliable AI features to production. At Proximity, you won’t just test software - you’ll help define how AI systems are validated, evaluated, and trusted at scale.

Responsibilities

• Build and maintain automation test suites for AI APIs, agentic workflows, and RAG-based systems.
• Implement LLM evaluation tests, covering accuracy, consistency, determinism, and performance characteristics.
• Collaborate closely with ML, backend, and full-stack engineering teams to validate releases and ensure production readiness.
• Contribute to regression, reliability, and performance testing across AI and platform services.
• Design test strategies for non-deterministic AI behavior, including guardrails, fallback paths, and quality thresholds.
• Continuously improve test coverage, tooling, and automation frameworks for AI-driven systems.
• 5+ years of experience in QA automation or quality engineering roles.
• Strong hands-on experience with Python and/or JavaScript, and modern testing frameworks.
• Experience building and maintaining API-level and integration test suites.
• Ability to design robust tests for complex, distributed systems.
• Familiarity with LLM- or ML-based systems and their unique testing challenges is strongly preferred.
• Strong analytical and problem-solving skills, with attention to detail.
• Ability to work independently while collaborating effectively with cross-functional teams.

Desired Skills

• Experience testing RAG pipelines, vector search, or AI agent workflows.
• Exposure to evaluation frameworks for LLMs or ML models.
• Familiarity with CI/CD integration for automated test execution.
• Understanding of performance and load testing for backend services.

Best in class compensation: We hire only the best, and we pay accordingly.

High-impact role: Help define quality standards for production AI systems.

Proximity Talks: Learn from experts across AI, engineering, and product.

Continuous learning: Grow your skills at the intersection of QA, automation, and AI.