Senior Data Scientist Generative AI
Share
Job Location:
San Mateo, CA - USA
Monthly Salary:
Not Disclosed
Posted on:
5 days ago
Vacancies:
1 Vacancy
Job Summary
WHY DATA SCIENCE & ANALYTICS
The Data Science & Analytics organizations mission is to increase our speed frequency and acumen in making decisions at scale by instilling a data-influenced approach to building products. We cover a wide area of the data spectrum including analytical data engineering product analytics experimentation causal inference statistical modeling and machine learning. Aligned and partnered with product verticals we use this extensive tool belt to discover new opportunities and unmet use cases influence and craft the product roadmap and prioritize build data products and measure impact on our community of players and developers.
WHY GENERATIVE AI
The Foundation AI groups mission is to enable Roblox Creators to accelerate their workflows and bring GenAI capabilities to millions of users. We envision a future where experiences on Roblox leverage generative text and speech to enable new interactions and generative 3D and 4D capabilities to empower new creative workflows and user experience.
As a Data Scientist on the team you will design build and operationalize evaluation for GenAI systems and work with cross-functional teams to improve model performance and the AI data generation flow. Since AI evaluation is core to GenAI safety quality and iteration speed we are building rigorous and scalable human and model-based evaluation systems that guide product decisions and model improvement. Youll combine annotation analysis design of experiments causal inference product analytics and model-based evaluation methods (such as LLM-as-a-judge / VLM-as-a-judge) to measure quality safety and user satisfactionand translate these findings into model and product improvements. Youll also help develop groundbreaking methodologies and tools that advance AI evaluation at Roblox and set industry standards. Beyond AI evaluation we proactively explore opportunities and solutions to improve the AI model and data generation flow.
Additionally we will build agentic workflows and AI agents for data solutions that enable teams to effectively access data extract data insights follow best practices and make data-informed decisions.
If you are a self-starter who is curious rigorous and passionate about building innovative solutions that deliver real business valueand thrive in a dynamic collaborative environmentthis role is for you.
You Will:
- Develop and improve evaluation frameworks for GenAI features (text image 3D 4D agentic workflow) including eval experiment design eval dataset design label reliability analysis results analysis and online evaluation based on user behavior and feedback.
- Establish best practices and guidelines for GenAI evaluation.
- Conduct product analytics online experiments (A/B tests) and causal analyses to quantify GenAI feature impact and identify opportunities.
- Build automated evaluation systems such as research and implement LLM-as-judge and VLM-as-judge methods.
- Research and apply state-of-the-art methodologies in GenAI evaluation.
- Advance reproducible evaluation tooling to lift evaluation rigor and efficiency at the company.
- Proactively explore and develop solutions to improve the AI model and data generation flow ensuring high-quality input for training and deployment.
- Design and implement agentic workflows and AI agents to enable teams to effectively access data extract data insights and follow best data practices.
- Partner closely with cross-functional teams to align goals plans and execution.
You Have:
- An advanced Degree and/or PhD in Statistics Economics Operations Research Computer Science Applied Math Physics Engineering or another quantitative field.
- 5 years of experience in data science or a related field.
- Familiarity with GenAI models and GenAI evaluation methods.
- Passion for the GenAI field and enthusiasm for continuously improving methods and practices to drive product quality and business impact.
- Ability to effectively use AI tools to enhance productivity in research ideation coding and documentation.
- Strong learning agility experience conducting applied research or writing technical papers is a plus.
- Proficiency in SQL Hive or Spark for transforming and manipulating large datasets.
- Experience with scripting languages such as Python or R.
- A demonstrated track record of solving open-ended data science and modeling problems that drive business impact and improve user experience.
Required Experience:
Senior IC
Key Skills
- Laboratory Experience
- Mammalian Cell Culture
- Biochemistry
- Assays
- Protein Purification
- Research Experience
- Next Generation Sequencing
- Research & Development
- cGMP
- Cell Culture
- Molecular Biology
- Flow Cytometry
About Company
Roblox is the ultimate virtual universe that lets you create, share experiences with friends, and be anything you can imagine. Join millions of people and discover an infinite variety of immersive experiences created by a global community!
