Research Engineer, Gemini AutoRater
Share
Job Location:
Mountain View, CA - USA
Monthly Salary:
Not Disclosed
Posted on:
Yesterday
Vacancies:
1 Vacancy
Job Summary
Snapshot
Advance research and engineering in large language models to improve AI feedback across Gemini (AutoRaters for evaluation Generative Reward models Self-critic capability of the core Gemini model).
At Google DeepMind we value diversity of experience knowledge backgrounds and perspectives and harness these qualities to create extraordinary impact. We are committed to equal employment opportunity regardless of sex race religion or belief ethnic or national origin disability age citizenship marital domestic or civil partnership status sexual orientation gender identity pregnancy or related condition (including breastfeeding) or any other basis as protected by applicable law. If you have a disability or additional need that requires accommodation please do not hesitate to let us know.
About us
Artificial Intelligence could be one of humanitys most useful inventions. At Google DeepMind were a team of scientists engineers machine learning experts and more working together to advance the state of the art in artificial intelligence. We use our technologies for widespread public benefit and scientific discovery and collaborate with others on critical challenges ensuring safety and ethics are the highest priority.
The role
The Gemini AutoRater team aims to push the research frontier of LLMs critiquing ability across various capabilities (Quality Coding Factuality Instruction Following etc). We work on AI feedback across Gemini evals and post-training specifically:
AutoRaters for evaluation
Generative reward models
Self-Critic and self-verification capability of Gemini models
These models are used across Gemini (e.g. OneRecipe).
You will be working alongside a world-class team of researchers and engineers to develop and advance the next generation of frontier AI models. Come work with us if you would like to pioneer work in this direction!
This role requires experience with LLM training and evaluation.
Key responsibilities
As a Research Engineer on the Gemini AutoRater team you will be at the forefront of evals and post-training research in Gemini. Typical responsibilities include:
Training AutoRaters and reward models (SFT and RL*F) evaluating them against human raters and deploying them in production (e.g. OneRecipe leaderboard).
Analyzing model error patterns and continuously improving them: collecting data new research ideas etc
As part of your role you will collaborate with numerous research and engineering teams working on Gemini models and their applications. You will deeply understand the relationships between the AutoRater workstream and other modeling capabilities. You will document and regularly present your work internally within the team and to senior stakeholders.
About you
In order to set you up for success as a Research Engineer at Google DeepMind we look for the following skills and experience:
Degree in machine learning statistics or related fields.
Strong hands-on experience with LLMs and foundation models
Strong end-to-end system building and prototyping skills.
In addition the following would be an advantage:
Self-directed researcher who can drive new research ideas from conception experimentation to productionization in a rapidly shifting landscape.
Experience with internal ML frameworks such as Gemax Scale/Prod and the Evergreen ecosystem.
A track record on landing research impact within multi-team collaborative environments under senior stakeholders.
Required Experience:
IC
Key Skills
- Robotics
- Machine Learning
- Python
- AI
- C/C++
- OS Kernels
- Research Experience
- Matlab
- Rust
- Research & Development
- Natural Language Processing
- Tensorflow
About Company
Artificial intelligence could be one of humanity’s most useful inventions. We research and build safe artificial intelligence systems. We're committed to solving intelligence, to advance science and benefit humanity.
