Data Scientist Survey Design, Data Annotation, and Machine Learning Evaluation

Apple is where individual imaginations gather together committing to the values that lead tongreat work. Every new product we build service we create or experience we deliver is thenresult of us making each others ideas stronger. The diversity of our people and their thinkingninspires the innovation that runs through everything we do. When we bring everybody in wencan do the best work of our lives. Here youll do more than join something youll addnsomething.

The Special Projects team at Apple is developing novel user-facing conversational features thatnleverage the multimodal capabilities of state-of-the-art foundation models. As part of thisnprocess we generate real-world and simulated data gather human data annotations analyzenthe results and use them to build and evaluate Large Language Model judges. We are lookingnfor a skilled Data Scientist to join our Machine Learning Evaluations teams. This person willnwork closely with ML Engineers to manage and analyze our human and automated datanannotation processes and to develop test and refine LLM judges for generative AI modelnevaluation. A successful candidate is experienced in survey design data annotation LLMnprompt engineering and prompt optimization and has strong statistical analysis skills.

Work closely with ML Engineers to understand data annotation needsn Design and manage data annotation processes including the development of userninstructions annotation pipeline processing and process improvementn Develop LLM auto-judges and judging criteria for generative AI model evaluationn Analyze collected data annotations to assess and refine LLM auto-judges

BA or Masters degree in Data Science Statistics or a quantitative social science fieldn 2 years of hands-on experience working in survey design and human data annotationn Proficiency in Pythonn Excellent communication skills

PhD in Data Science Statistics or a quantitative social science fieldn Hands-on industry experience with product-focused statistical analysisn Experience working with large-scale multimodal data and data-annotation pipelinesn Experience with LLM prompt engineering u0026 prompt optimizationn Experience with LLM auto-judges for generative AI model evaluationn A track record of publications or technical presentations in Data Science or a related fieldn Excellent at cross-functional collaboration