Sr Machine Learning Engineer, Tech Lead — Autograder Systems, Evaluation

We are looking for a Senior MLE Tech Lead to join a centralized evaluation organization and define the next generation of autograder quality across 20 of Apples most visible generative AI features. You will own the end-to-end technical vision for how we evaluate model outputs at scale pioneering state-of-the-art methods raising the technical bar and leading a team of talented MLEs to build a robust autograder training and hillclimbing system from the ground is a high-impact hands-on leadership role at the intersection of model evaluation data quality and ML systems engineering. You will work closely with model developers data teams and product partners to ensure our autograders are fast accurate and continuously improving directly shaping the quality of AI experiences used by hundreds of millions of people.

In this role you will focus on:nnTechnical Leadershipnn* Define and drive the technical roadmap for autograder quality researching and introducing novel methods such as reward modeling LLM-as-judge preference learning and calibration techniques to measurably improve evaluation accuracy.n* Architect and lead the build-out of a scalable autograder training pipeline encompassing data curation model fine-tuning evaluation harnesses and versioning.n* Design and own the hillclimbing system that iteratively improves autograder performance through systematic prompt and model optimization loops.n* Establish quality benchmarks confidence metrics and failure analysis frameworks that enable the team to track trust and act on autograder u0026 Collaborationnn* Mentor and technically guide a team of MLEs through design reviews modeling standards and hands-on problem-solving fostering a culture of rigor and continuous learning.n* Partner with data annotation teams to define labeling guidelines that feed autograder training.n* Collaborate with feature engineers to align autograder signals with broader training and product objectives.n* Translate complex technical trade-offs into clear narratives for engineering product and leadership audiences.

Masters or PhD in Computer Science Machine Learning Artificial Intelligence or a related field.n5 years of industry experience in machine learning with a strong focus on LLM or VLM expertise in prompt-tuning and fine-tuning techniques (SFT RLHF DPO or equivalent) with proven experience of model calibration and uncertainty with data flywheel design leveraging model outputs to continuously improve future training in Python and ML frameworks (PyTorch preferred).

Strong ML systems instincts you care deeply about data quality reproducibility latency and in human-in-the-loop annotation pipelines and inter-annotator agreement experience on an evaluation infrastructure or model quality team.