Annotation Data Scientist, Evaluation Integrity (Siri)

Join the team redefining what a deeply personal and integrated assistant can be. nnAs part of the Siri organization you will help shape one of the worlds most widely used AI assistants powered by our next-generation of Apple Intelligence with capabilities like personal context understanding and on-screen awareness built with privacy from the ground up. Your work will have direct meaningful impact for users across iOS iPadOS macOS watchOS and is a rare opportunity to build at the intersection of cutting-edge AI and human-centered design shipping technology that is centered around users and their needs.

Play a part in the ongoing revolution in human-computer interaction. Siri is evolving and the way we evaluate it has to evolve with it. Join the Evaluation Integrity team to help build the trusted quality signal behind every Siri the Siri evaluation organization the Human Evaluation sub-team is responsible for answering the question: can we trust our evals We do that by designing human-in-the-loop (HITL) annotation tasks that scrutinize every moving part of an agentic evaluation the simulated user agent the conversation it has with Siri and the automated evaluators that grade the exchange. This role sits at the intersection of data science human annotation engineering and evaluation methodology and is instrumental in turning human judgment into a rigorous reproducible signal that directly informs pre-ship model and product an Annotation Data Scientist on the Evaluation Integrity team you will design and run HITL annotation projects that evaluate the quality and authenticity of agentic user personae the validity of agent-to-agent conversations and the reliability of LLM-as-judge and rule-based evaluators against Siris product specifications. You will own annotation initiatives end-to-end; from rubric design and tooling through annotator calibration to data science analysis that turns annotator judgments into actionable signal for modeling planning and product teams.n

Design HITL annotation tasks for agentic evaluation. Advise on rubrics and design workflows that ask annotators to assess (a) the quality and authenticity of user agent personae (b) the validity of agent-to-agent conversations and (c) whether agentic evaluators verdicts align with Siris product specifications and human interface maintain and iterate on annotation guidelines. Translate evolving Siri capabilities and product specs into clear defensible rubrics for human grading aligned with agentic evaluators; run calibration sessions; monitor inter-annotator agreement; and refine guidelines based on edge cases surfaced during multiple annotation programs in parallel. Plan scope and manage human evaluation tasks end-to-end requirements gathering annotator coordination vendor management timeline tracking and stakeholder custom annotation tooling in partnership with software engineers. Prototype task UIs specify tool requirements and collaborate with tooling engineers on the annotation platforms the Human Evaluation team relies data science rigor to human-labeled data. Use Python to build analysis pipelines that measure evaluator accuracy against the annotator pool surface discrepancies between LLM-judge and rule-based evaluators and quantify the reliability of each agentic evaluator as a source of annotator feedback into evaluator improvements. Close the loop between annotators and the data scientists and software engineers who own user agents and automated evaluators feeding findings back into prompts rubrics and product to the organization-wide eval health story. Partner with the User Feedback and Eval Science sub-team to ensure human signal is represented in the eval health report delivered to leadership.n

Bachelors or Masters degree in a quantitative or related field such as Data Science Computer Science Linguistics Statistics or Cognitive Science or equivalent job-related experience.n5 years of hands-on experience working with human-annotated datasets or human-in-the-loop evaluation methodologies for machine learning natural language processing or large language model systems.n5 years of experience using Python for data processing analysis and prototyping including experience with libraries such as pandas Jupyter and at least one data visualization designing implementing and communicating annotation schemas rubrics or ontologies for machine learning training or evaluation managing multiple concurrent dataset curation efforts including scoping work iterating on guidelines coordinating with in-house or vendor annotators and monitoring annotator performance metrics such as accuracy throughput and inter-annotator specifying or designing custom annotation tooling in collaboration with software engineers.

Experience evaluating LLM-powered or agentic systems including familiarity with LLM-as-judge methodologies rubric-based grading or trajectory and tool-call with statistical methods that address accuracy and variability in human annotation data such as inter-annotator agreement Cohens or Fleiss kappa Krippendorffs alpha or -querying experience with SQL Spark or similar and comfort working with large complex real-world building pre-ship evaluation pipelines for conversational or assistant with prompt engineering or with designing simulated user personae for agent running annotation programs across multiple locales or at large written and verbal communication skills with the ability to explain technical topics clearly to data scientists engineers annotators and cross-functional ability to collaborate effectively across functions and drive projects of varying sizes and scopes knowing when to dive deep and when to delegate.