🤖 15+ AI Agents working for you. Find jobs, score and update resumes, cover letter, interview questions, missing keywords, and lots more.
Cookies & analytics consent
We serve candidates globally, so we only activate Google Tag Manager and other analytics after you opt in. This keeps us aligned with GDPR/UK DPA, ePrivacy, LGPD, and similar rules. Essential features still run without analytics cookies.
Read how we use data in our Privacy Policy and Terms of Service.
🤖 15+ AI Agents working for you. Find jobs, score and update resumes, cover letter, interview questions, missing keywords, and lots more.
Ncs • Sydney, New South Wales, Australia
Role & seniority: Senior Test Analyst (lead testing for AI-driven, multi-system solutions)
Stack/tools: API/integration testing; multi-system orchestration; asynchronous workflows; Python and shell scripting; LLM behavior, agent architectures (function/tool calling, planners, RAG, memory, guardrails); automated test suites; audit-ready test evidence
Design, automate, and execute testing for LLM-powered applications and agentic systems across DEV, SIT, UAT, and PROD
Lead Responsible AI testing (safety, guardrails, fairness, explainability), red-teaming/adversarial evaluation, and integration testing
Produce audit-ready test evidence and documentation for Cyber and Change boards; coordinate cross-functional releases and post-deployment checks
Strong API and integration testing across distributed, multi-system workflows
Deep understanding of LLM behavior and agent architectures
Hands-on automation in Python and shell scripting
Responsible AI testing knowledge (safety categories, content moderation, hallucination detection, explainability, data/privacy)
Experience presenting test evidence to Cyber/Security and Change boards; release coordination across DEV/SIT/UAT/PROD
Experience with red-team/adversarial testing, prompt injection risks, guardrail weaknesses
Knowledge of traceable RAG, data governance, and evaluation metrics
Location & work type: Australia-based; full-t
We are NCS Australia! We have brought together Australia’s best tech talent, working as one to harness technology and advance our communities, to make the extraordinary happen.
We're proud of our rich history of partnering with our clients to develop end to end digital technology solutions to accelerate their transformation and innovation journey. For over 40+ years our diverse global teams have delivered large-scale, mission-critical, multi-platform projects to governments and enterprises.
Job Description
We are seeking a Senior Test Analyst to design, automate, and execute testing for LLM‑powered applications and agentic systems operating across multiple platforms and services. You will lead Responsible AI testing (safety, guardrails, fairness, explainability), red‑teaming/adversarial evaluation, and multi-system integration testing. You will also coordinate releases across DEV, SIT, UAT, and PROD, producing audit‑ready test evidence for Cyber and Change boards.
You’ll partner closely with AI engineering, platform, SecOps, and Network Operations teams to assure safety, reliability, and compliance of AI capabilities at scale.
As a Senior Test Analyst, you will design and execute comprehensive test strategies for AI‑driven and multi‑system solutions, ensuring quality across integration, API, functional, and Responsible AI domains. You will develop automated test suites using Python or shell scripting, validate LLM behaviour and agent workflows, and assess system reliability, safety, and explainability through structured test scenarios. You will contribute to red‑team and adversarial testing to identify vulnerabilities such as prompt injection, misuse of agent tools, and guardrail weaknesses. The role requires leading testing across multiple environments (DEV, SIT, UAT, PROD), preparing high‑quality test evidence and documentation for Cyber and Change Board reviews, and coordinating with cross‑functional teams to ensure smooth releases, prompt defect resolution, and adherence to governance standards.
Qualifications
Strong API & integration testing across distributed systems; experience with multi‑system orchestration and asynchronous workflows. Demonstrated understanding of LLM behavior and agent architectures (function/tool calling, planners, RAG, memory, guardrails). Hands‑on automation in Python and shell
Responsible AI testing knowledge: safety categories, content moderation, hallucination detection, explainability (e.g., traceable RAG), data/privacy considerations, evaluation metrics. Experience preparing and presenting test evidence and sign‑off documentation to Cyber/Security and Change boards. Proven track record coordinating release testing across DEV/SIT/UAT/PROD, including cutovers and post‑deployment checks.
Additional Information
NCS Australia is where you can feel at home, nurturing your talents and skills as we make tomorrow together, one day at a time. Our benefits include paid parental leave, initiatives focused on your well-being and discounted health insurance. You will also enjoy discounts on various products and services and be regularly recognised and rewarded for high performance. We are committed to your career development through our Capability Fingerprint, industry and partner training programs, special interest groups, and an AI-driven learning platform. No matter where you are in your career, we offer meaningful work and opportunities for growth.
NCS Australia is an equal-opportunity employer, and we take pride in our commitment to valuing and supporting our people and the communities we serve.We are dedicated to attracting, retaining and developing our people regardless of gender identity, ethnicity, sexual orientation, disability and age. Applications are encouraged from all sectors of the community and we strongly encourage applications from the Veterans, Aboriginal and/or Torres Strait Islander community.
At NCS Australia, we are committed to supporting adjustments throughout the recruitment and selection process, as well as during employment. We actively support and encourage people with disability to apply.
We’ve got this. We request that you do not contact NCS employees outside of the Talent Acquisition team. NCS exclusively accepts resumes from agencies on our preferred supplier panel through the NCS Agency Portal. Agencies that submit resumes must have a valid fee agreement and be assigned to the particular requisition by the Talent Acquisition team. Any resumes that are submitted outside of this process will become the sole property of NCS. If a candidate is hired outside of this process, no fee or payment will be given.
To be eligible for a position with us, applicants will need to have valid work rights for Australia and be willing to undergo a comprehensive background checking process, including probity and police checks