ML Research Engineer, AI Evaluation Platform

AI systems are only as trustworthy as the methods used to evaluate them. At Apple where AI powers experiences for billions of people getting evaluation right is not a support functionit is a foundational science. Our team part of Apple Services Engineering is building that scientific foundation: rigorous scalable evaluation methodology for LLMs agentic systems and human-AI makes this team unusual is its interdisciplinary core. You will work alongside measurement scientists (psychometrics validity theory) ML researchers and platform engineersbringing together ML research statistical rigor and production engineering. We are looking for an ML Research Engineer who can move fluidly across this landscape: someone who loves implementing the latest techniques in AI has the engineering instincts to make them robust and scalable and thrives at the intersection of research and production.

This is a combined research and engineering role sitting with and between research/applied scientists and platform engineers. New evaluation research can be challenging to use at scalethats where your skills in both machine learning and engineering come into the research side you will partner with scientists to rapidly prototype their ideas implement methods from recent papers run large-scale experiments and provide critical feedback grounded in your engineering experience. On the engineering side you will work with platform engineers to bring those research prototypes into productionmoving from Python packages on local machines to robust services deployed in the past experience in research is not required a desire to advance the state of the art in AI evaluation is. You should be ready to jump in across the full lifecycle of bringing new research into production at scale speaking both the language of research and the language of engineering.

Rapid Prototyping u0026 Experimentation: Collaborate with research and applied scientists to translate evaluation research ideas into working prototypesimplementing methods from recent papers building experimental pipelines and iterating quickly to validate hypotheses in areas such as preference learning LLM-as-judge calibration and automated failure -to-Production Bridge: Own the lifecycle of moving evaluation methods from research prototypes to production-ready systems. Refactor research code into robust well-tested Python packages and partner with platform engineers to deploy them as scalable services APIs and SDK Infrastructure: Design and maintain the infrastructure for running large-scale evaluation experimentsorchestrating LLM judge calls managing datasets tracking experiment results and ensuring reproducibility across the teams research Feedback u0026 Collaboration: Serve as a critical technical partner to researchers providing engineering perspective on feasibility scalability and system design. Identify opportunities where engineering improvements (parallelization caching smarter batching) can unlock new research directions or dramatically accelerate Evaluation Methods: Identify bottlenecks in evaluation workflows and engineer solutions to operate at Apple scaleoptimizing for throughput cost and reliability when running evaluation methods across large model populations and diverse use Quality u0026 Engineering Standards: Champion engineering best practices within the research workflow including version control automated testing documentation and CI/CD raising the bar for code quality across the research-engineering -Functional Integration: Work across the research and platform engineering teams to ensure that evaluation methods integrate seamlessly with Apples broader ML infrastructure developer workflows and internal tooling ecosystem.

Bachelors degree in Computer Science Machine Learning Software Engineering or a closely related field (Masters preferred)n2 years of hands-on experience in a role combining machine learning and software engineering (e.g. ML engineer research engineer or applied scientist with strong engineering output) or a Masters degree in Computer Science Machine Learning or a closely related field with relevant project experiencenStrong proficiency in Python and the modern ML ecosystem (PyTorch JAX or TensorFlow) with demonstrated ability to implement complex methods from recent ML papersnSolid software engineering fundamentals: clean code design version control testing debugging and performance optimizationnExperience working with large language modelswhether fine-tuning inference prompting pipelines or building LLM-powered applicationsnDemonstrated ability to work across the research-to-production spectrum: you have taken experimental or prototype code and made it robust scalable and usable by othersnPractical experience with cloud-native development and deployment: containerization (Docker/Kubernetes) CI/CD pipelines and distributed computing frameworks (e.g. Ray Spark)nStrong communication skills and comfort working in interdisciplinary teams with the ability to engage productively with both researchers and platform engineersnComfort with ambiguity and new problem spacesyou thrive when building something that doesnt yet have a playbook

Masters or Ph.D. in Computer Science Machine Learning or a related fieldnExperience with evaluation-specific methods or frameworks: LLM-as-judge approaches reward modeling RLHF calibration techniques benchmark design or human evaluation methodologynFamiliarity with modern evaluation tools and frameworks (e.g. DeepEval Ragas TruLens LangSmith) and an understanding of how to implement and scale model-based evaluation workflowsnTrack record of contributing to research outputsco-authored publications open-source contributions or internal research reportseven if research is not your primary rolenExperience with the engineering challenges specific to generative AI and agentic systems: managing token economics handling non-deterministic outputs evaluating multi-turn agent trajectories and tool usagenFamiliarity with statistical concepts relevant to evaluation: calibration inter-rater reliability scoring rules or measurement validitynExperience in fast-moving early-stage teams where you helped define technical direction and engineering culture from the ground up