🤖 15+ AI Agents working for you. Find jobs, score and update resumes, cover letter, interview questions, missing keywords, and lots more.
Cookies & analytics consent
We serve candidates globally, so we only activate Google Tag Manager and other analytics after you opt in. This keeps us aligned with GDPR/UK DPA, ePrivacy, LGPD, and similar rules. Essential features still run without analytics cookies.
Read how we use data in our Privacy Policy and Terms of Service.
🤖 15+ AI Agents working for you. Find jobs, score and update resumes, cover letter, interview questions, missing keywords, and lots more.
Talent • Melbourne, Victoria, Australia
Role & seniority: Senior Test Automation Engineer (AI/ML/LLM/GenAI), Technology Delivery team
Languages: Python (advanced), FastAPI, pytest
UI/API/Automation: Playwright, Selenium, Appium, REST Assured
AI/ML: Azure OpenAI, AWS Bedrock, Databricks, Elasticsearch, Vector Search
Frontend/Backend: React/Next.js, APIs, distributed services
Cloud/DevOps: AWS, Azure, Git, CI/CD pipelines
Design and implement enterprise-grade test automation frameworks across UI, API, backend, and AI services; build end-to-end regression and performance suites
Develop automated testing strategies for ML/LLM/Generative AI applications; validate AI outputs for accuracy, relevance, and response quality
Integrate automation into CI/CD; monitor metrics, drive risk-based testing, and mentor junior engineers; collaborate with developers, product teams, and stakeholders
Proven experience building scalable test automation frameworks
Hands-on testing of AI/NLP/LLM-based applications; complex distributed systems, APIs, data platforms
Advanced Python development (FastAPI, pytest) and UI test automation (Playwright)
Experience with cloud-based testing environments, CI/CD integration, and producing meaningful quality metrics
Certifications: ISTQB Advanced Test Automation Engineering (CTAL-TAE), AI/ML/Data Science
Experience with MLflow or Opik, vector databases, search techn
We are seeking an experienced Senior Test Automation Engineer (ML, LLM, GenAI) to join our Technology Delivery team supporting a strategic AI-powered system. This role plays a critical part in ensuring the quality, reliability, and performance of modern Machine Learning, Large Language Model (LLM), and Generative AI solutions. You will design and implement scalable automation frameworks across the full technology stack - from React-based user interfaces through to backend services, AI models, search platforms, and data orchestration pipelines. You will lead testing innovation for AI-driven applications, validating contextual accuracy, factual correctness, and ongoing performance of LLM models while enabling continuous delivery of platform enhancements.
Key Responsibilities
Design and implement enterprise-grade test automation frameworks across UI, API, backend, and AI services Develop automated testing strategies for ML, LLM, and Generative AI applications Validate AI outputs for accuracy, relevance, and response quality across multiple models Integrate automation into CI/CD pipelines to support continuous delivery Build end-to-end regression and performance test suites Monitor automation metrics and provide actionable quality insights Lead test planning, coverage strategy, and risk-based testing activities Mentor junior engineers and promote automation best practices Collaborate closely with developers, product teams, and stakeholders Drive continuous improvement of testing frameworks and DevOps practices
Technical Environment
Languages & Tools: Python, pytest, FastAPI
Automation: Playwright, Selenium, Appium, REST Assured
AI & Data: Azure OpenAI, AWS Bedrock, Databricks, Elasticsearch, Vector Search
Frontend & Backend: React / Next.js, APIs, distributed services
Cloud & DevOps: AWS, Azure, Git, CI/CD pipelines
Skills & Experience
Strong experience designing scalable test automation frameworks Hands-on testing of AI, NLP, or LLM-based applications Advanced Python automation development Experience testing complex distributed systems, APIs, and data platforms Deep understanding of Agile delivery and DevOps practices Experience with cloud-based testing environments Ability to produce meaningful quality metrics and reporting
What You'll Bring
Technical leadership in automation and quality engineering Strong analytical and problem-solving capability Passion for emerging AI technologies and innovation Ability to work autonomously while contributing to high-performing teams
Desirable Certifications
ISTQB Advanced Test Automation Engineering (CTAL-TAE) AI/ML or Data Science certifications Python or SQL certifications
Why Join Us?
This is an opportunity to work at the forefront of AI and Generative AI technology, shaping testing practices for next-generation intelligent platforms while delivering meaningful enterprise impact. For more information or a confidential discussion about this role, please contact Ronald at ronald.tran@talentinternational.com
Desired Skills and Experience
*Advanced Python programming with expertise in Fast API, pytest, and data science libraries (Pandas, NumPy) *Strong experience building backend test automation frameworks using Python, preferably with Fast API *Experience testing Large Language Models, natural language processing systems, or AI-driven applications. *Demonstrated experience testing complex, multi-component systems including APIs, databases, and search technologies. *Proven ability to design and build scalable, reusable test automation frameworks from the ground up. *Experience with LLM observation and evaluation tools, such as MLflow or Opik
*Experience with cloud platforms: AWS services (S3, Aurora, Bedrock, Transfer), Azure services (OpenAI, Document Intelligence), and modern data platforms like Databricks *Experience testing Elasticsearch, vector databases, or similar search technologies. *Experience integrating test automation into CI/CD pipelines using Git and collaborative development workflows. *Significant hands-on experience with UI test automation (such as Playwright), particularly with React/Next.js applications