🤖 15+ AI Agents working for you. Find jobs, score and update resumes, cover letter, interview questions, missing keywords, and lots more.
Cookies & analytics consent
We serve candidates globally, so we only activate Google Tag Manager and other analytics after you opt in. This keeps us aligned with GDPR/UK DPA, ePrivacy, LGPD, and similar rules. Essential features still run without analytics cookies.
Read how we use data in our Privacy Policy and Terms of Service.
🤖 15+ AI Agents working for you. Find jobs, score and update resumes, cover letter, interview questions, missing keywords, and lots more.
Matlen Silver • United States
Role & seniority: AI Quality Assurance Engineer – Senior Advisor II (senior individual contributor)
Stack/tools: Python; AI QA methodologies; ML lifecycle tools (MLflow or similar); Retrieval-Augmented Generation (RAG) frameworks (e.g., LangChain); data validation; model performance testing; bias/fairness evaluation; auditability/Explainability; continuous monitoring; probability/probabilistic coding concepts
Validate data integrity and quality across AI/ML pipelines; assess model performance against defined metrics (accuracy, robustness, scalability, reliability)
Conduct bias and fairness evaluations; ensure auditability and explainability to meet regulatory requirements
Implement continuous monitoring with escalation and human-in-the-loop oversight; design/test RAG workflows; develop automated functional/robustness tests
15+ years as senior software quality engineer or technical IC
3–5+ years hands-on with modern AI tools in production; experience in enterprise AI
Proficiency in Python; strong coding/scripting for automation and testing
Expertise in AI QA (data validation, model performance testing, bias/equity evaluation, explainability)
Experience with ML lifecycle/monitoring tools (MLflow or similar)
Knowledge of AI governance, compliance, auditability; excellent cross-functional communicator
Bachelor’s or Master’s in CS/Data Science or related field
Experien
Job Title: AI Quality Assurance Engineer – Senior Advisor II
Remote: Yes
Contract To Hire: Yes
Compensation: $120 - $150/Hour W2 (Full Time Conversion $225K - $300K/Year)
Location: Onsite
Employment Type: Contract / Contract to Hire (as applicable) About the Opportunity We are building a brand-new AI Governance and Quality Engineering team from the ground up to support a large-scale enterprise AI initiative. This team will establish guardrails and governance standards for how AI is implemented across the software development lifecycle, ensuring secure, compliant, and high-quality AI adoption at scale. This is a senior-level individual contributor role suited for someone with deep experience in AI/ML systems who can operate as a strategic advisor. The ideal candidate brings strong technical expertise, exceptional communication skills, and the ability to influence engineering standards across cross-functional teams. Key Responsibilities
Validate data integrity and quality across AI/ML pipelines to ensure reliable model inputs
Assess model performance against defined success metrics, including accuracy, robustness, scalability, and reliability
Conduct bias and fairness evaluations to maintain ethical and compliant AI outcomes
Ensure auditability and explainability of AI models to meet regulatory and transparency requirements
Implement continuous monitoring strategies, including escalation protocols and human-in-the-loop oversight for critical AI decisions
Design and test Retrieval-Augmented Generation (RAG) workflows for precision, relevance, and consistency
Develop and maintain automated functional and robustness tests for AI components (strong coding required)
Leverage AI QA tools and frameworks (e.g., MLflow or similar) to validate and monitor AI pipelines
Define and track enterprise-level AI quality, reliability, and compliance metrics
Act as an internal and external consultant to guide AI governance, guardrails, and quality standards across development teams
Contribute to the integration of AI layers into existing development workflows, identifying process simplification and efficiency gains while staying within security and compliance boundaries
Required Qualifications
15+ years of experience as a senior-level software quality engineer or technical individual contributor
Minimum 3–5 years of hands-on experience working directly with modern AI tools and AI tool implementation in production environments
Prior experience leveraging AI/ML capabilities in enterprise systems
Strong experience with Retrieval-Augmented Generation (RAG) workflows (e.g., LangChain or similar frameworks)
Experience working with probabilistic systems and probabilistic code concepts
Deep expertise in AI QA methodologies, including data validation, model performance testing, bias and fairness evaluation, and explainability
Strong coding and scripting proficiency (Python strongly preferred) for automation and testing
Hands-on experience with ML lifecycle and monitoring tools such as MLflow or comparable platforms
Strong understanding of AI governance, regulatory requirements, auditability, and compliance standards
Excellent communication skills with the ability to operate as a high-level advisor across technical and business stakeholders
Preferred Qualifications
Bachelor’s or Master’s degree in Computer Science, Data Science, or related technical discipline
Experience influencing enterprise AI standards and best practices
Background working in highly regulated environments
This is a highly visible opportunity to shape AI governance and quality standards at the enterprise level while helping scale responsible AI adoption across the organization.