Apple logo

Director, Evaluation and Quality Assurance, Services Special Projects

Apple • Cupertino, California, United States

onsite
Posted Oct 15, 2026

**Role & seniority: ** Director (first 0-to-1, first key hire) — build and lead Evaluation & Quality Assurance function

**Stack/tools: **

  • LLM-based autograders, rubric/rubric design, evaluation set development/validation/calibration, human-model alignment

  • Automated grading + human evaluation programs

  • Logging/instrumentation for AI quality measurement

  • A/B testing, QA for third-party API integrations

  • Conversational AI / ranking / recommendation / search / agent/tool-use systems; integrations incl. external partner APIs and financial systems

  • Top 3 responsibilities:

    1. Define end-to-end “quality” criteria and metrics across product layers (triggering, relevance/ranking, transaction correctness, integrations).

    2. Build evaluation + QA infrastructure (automation, grading, evaluation sets, human evaluation) with continuous, always-on pipelines and release-linked pass/fail gates.

    3. Establish the team, tooling, and operating rhythm; drive proactive issue detection and influence product/shipping decisions with engineering & ML leadership.

  • Must-have skills:

    • 12+ years building/leading QA/evaluation in technical product environments; 0-to-1 function creation (metrics/process/tooling/org).

    • Deep expertise in LLM evaluation (rubrics, calibration, validation, alignment) and using it to drive improvements.

    • Human evaluation methodology, logging, and A/B testing at consumer-product scale.

    • Experience evaluating con

Full Description

We are looking for a Director to build and lead the evaluation and quality assurance function for one of our most ambitious multi-year engineering

initiatives. This is a true 0-to-1, first-key-hire role: There is no existing team, tooling, or process to inherit. You will define how we measure quality across an entire product end to end, build the automation and evaluation infrastructure to do it at scale (or borrow other teams' infrastructure), and build and lead the organization that runs it.

DESCRIPTION

You will own evaluation and quality assurance for every layer of the product — from whether the system is being correctly triggered for the right set of user requests, through whether it surfaces and ranks the right results, to whether the underlying transactions execute correctly, including integrations with external partner APIs, and financial systems. Quality here isn't a single test suite — it spans conversational understanding, relevance and ranking, and revenue-bearing correctness, all against a large, constantly evolving real-world dataset. You will create and drive the evaluation and quality strategy and implementation for the product. This will extensively utilize automated grading systems and human evaluation programs where appropriate. You will make the calls about where to invest in each. You will define what "correct" and "good" mean for this product, wire that definition into continuous, always-on evaluation pipelines with clear pass/fail quality gates tied to releases, and build the team, tooling, and operating rhythm to catch issues before they reach users. As the first hire in this space, you will operate with significant latitude and ambiguity — setting strategy, process, and technical direction with minimal existing scaffolding, while also building and scaling an organization. You will partner closely with engineering, applied ML, and program management leadership, including senior executive stakeholders, and you will be a key voice in defining what "done" and "shippable" mean for a product still taking shape, as well as iterating on decision points in production after the product ships.

MINIMUM QUALIFICATIONS

12+ years of experience in data science, machine learning evaluation, quality assurance, or test engineering roles in a technical product environment, including experience building and leading multi-level QA or evaluation organizations through managers. Demonstrated experience building an evaluation or QA function from the ground up — defining metrics, process, tooling, and organizational structure where none previously existed. Extensive experience utilizing LLM-based autograders and rubrics, including rubric design, evaluation set development, validation, calibration, and human-model alignment, and using them to proactively drive product and model improvements. Deep experience in traditional human evaluation methodology, logging, and A/B testing for consumer-facing products at scale. Experience evaluating conversational AI, recommendation, ranking, or search systems, and — ideally — agentic or tool-use systems that take real-world actions, not just generate responses. BS in Computer Science, Engineering, or a related technical field, or equivalent experience. Experience with quality assurance for third-party API integrations. Experience with logging infrastructure and instrumentation for AI product quality measurement. Exceptional written and verbal communication skills, with a track record of presenting quality status, risk, and trade-offs clearly to, and influencing decisions among, senior executive leadership.

PREFERRED QUALIFICATIONS

15+ years of experience in data science, machine learning evaluation, quality assurance, or test engineering roles in a technical product environment, including experience building and leading multi-level QA or evaluation organizations through managers.

Quality AssuranceMachine Learning EvaluationData ScienceTest EngineeringLLM-based AutogradersRubric DesignHuman EvaluationA/B TestingConversational AIRecommendation SystemsRanking SystemsAPI IntegrationsLogging InfrastructureInstrumentationStrategic LeadershipTechnical Directionmulti-location

Cookies & analytics consent

We serve candidates globally, so we only activate Google Tag Manager and other analytics after you opt in. This keeps us aligned with GDPR/UK DPA, ePrivacy, LGPD, and similar rules. Essential features still run without analytics cookies.

Read how we use data in our Privacy Policy and Terms of Service.