ML Evaluation Specialist, Human Data

At Apple we dont just build products we build experiences fueled by world-class data. ThenHuman-centered AI team within Apple Services Engineering is looking for an ML Evaluation Specialist Human Data to join our Data Quality and Operations division to spearhead complex multi-stakeholder operations that specialize in data collection curation annotation and human evaluation efforts across Apple Music App Store TV Podcasts and this role you will own the operational strategy and continuous improvement of large-scale multilingual human data programs from designing onboarding scaffolds that progressively build annotator calibration to analyzing annotator behavior patterns to identify where automation can offload low-judgment decisions to enforcing quality frameworks that close the loop between annotator struggle and task redesign. You will identify where human judgment is essential and where it could be better directed then build the scaffolding automation and feedback systems that let annotators focus their cognitive energy where it matters most. Because this work cuts across engineering data science research procurement and legal a critical part of the role is serving as the connective tissue between teams who each own a piece of this space aligning on shared standards surfacing gaps and ensuring that insights from the annotation layer inform upstream decisions about task design and tooling. You will bring a point of view on human data best practices and translate it into scalable human-centered approaches that make generative AI features safer and more ideal candidate brings a rare combination of technical depth and program execution skills. You are comfortable designing and deploying sophisticated data pipelines in the morning and then seamlessly transitioning to present comprehensive quality rectification strategies to stakeholders in the afternoon. You care deeply about data quality and human alignment have a creative and systematic approach to finding and fixing problems and find motivation in wide-ranging work whose impact shows up in everyday Apple experiences.

In this role you will own the operational strategy and continuous improvement of large-scale multilingual human data programs from designing onboarding scaffolds that progressively build annotator calibration to analyzing annotator behavior patterns to identify where automation can offload low-judgment decisions to enforcing quality frameworks that close the loop between annotator struggle and task redesign. You will identify where human judgment is essential and where it could be better directed then build the scaffolding automation and feedback systems that let annotators focus their cognitive energy where it matters most. Because this work cuts across engineering data science research procurement and legal a critical part of the role is serving as the connective tissue between teams who each own a piece of this space aligning on shared standards surfacing gaps and ensuring that insights from the annotation layer inform upstream decisions about task design and tooling. You will bring a point of view on human data best practices and translate it into scalable human-centered approaches that make generative AI features safer and more ideal candidate brings a rare combination of technical depth and program execution skills. You are comfortable designing and deploying sophisticated data pipelines in the morning and then seamlessly transitioning to present comprehensive quality rectification strategies to stakeholders in the afternoon. You care deeply about data quality and human alignment have a creative and systematic approach to finding and fixing problems and find motivation in wide-ranging work whose impact shows up in everyday Apple experiences.

Lead the end-to-end execution of human data collection programs for multilingual multimodal and multi-turn AI features from intake and scoping to delivery and retrospectivenEstimate and maintain project timelines capacity needs and cost while anticipating and proactively resolving bottlenecks to ensure timely executionnDesign and own the measurement framework for human data collection initiatives defining key indicators such as spend speed inter-rater reliability and volume and building reporting systems that surface actionable insights to stakeholdersnBuild and implement human data quality frameworks including developing statistical process controls and behavioral signals to proactively detect and remediate quality degradation throughout the annotation lifecyclenApply human-centered AI principles to influence data collection task design identifying and reducing sources of cognitive burden that impact human performance consistency and wellbeingnSystematically analyze data collection workflows to identify inefficiencies and scalability gaps then design and implement innovative workflows (e.g. agent- and machine-in-the-loop) and build the supporting data pipelines and ETL services to improve data quality and diversity while optimizing cost and lead timenPartner with Legal Privacy and New Product Security to design and implement compliant data collection and user study programs ensuring all human data workflows adhere to Apples privacy governance and regulatory standardsnDesign and own the data collection lifecycle with external and internal workforces including building onboarding and calibration programs performance frameworks and quality audits that ensure reliable high-quality deliverablesnAct as the connective tissue across engineering product legal security procurement and vendor teams ensuring alignment clear communication and follow-through

Bachelors degree or higher in Cognitive Science Linguistics or a related field that includes an experimental or empirical componentn4 years of experience defining and leading cross-team human data programs for AI/ML including annotation operations quality frameworks and evaluation strategies within an NLP/NLU or generative AI environmentnProficiency in programming and data languages (Python R SQL) to process analyze query large datasets extract insights automate tasks and monitor program performancenHands-on experience designing and managing 01 human-in-the-loop data collection annotation and evaluation initiatives including driving and incorporating agentic workflows to improve quality and scalabilitynExperience working with diverse data types (e.g. speech text multimodal) across multiple languagesnExpertise in end-to-end data annotation quality management including the ability to develop statistical process controls and data quality metricsnFamiliarity with privacy-preserving data handling practices and compliance frameworksnDemonstrated success optimizing data pipelines and workflows to improve quality reduce lead time and scale operationsnExperience working cross-functionally with engineering data science legal privacy and third-party suppliers

Masters degree or higher in Cognitive Science Linguistics or a related field that includes an experimental or empirical componentn2 years of experience owning data strategy for frontier AI development and evaluation with experience in human alignment methodologies and agentic GenAI systemsnExperience managing external vendor or workforce partners at scalenFamiliarity with AI Safety and Responsible AI principles including experience applying them to data collection or annotation workflowsnStrong organizational skills and execution-oriented mindset; ability to balance attention to detail with big-picture thinking in an environment where program scope and priorities evolve quicklynExcellent written and verbal communication skills; able to translate technical concepts for non-technical stakeholders