Research Scientist

Join a research and engineering team developing benchmarks and evaluation methods for advanced language models. Your work will help AI labs and enterprise teams assess model capabilities and make informed decisions before deploying models in production.

• Evaluate newly released AI models across a range of use cases and report on their capabilities.
• Build benchmarks from the ground up including dataset development and coordination with labelers.
• Advance automated evaluation methods for generated text to improve scalability and accuracy.
• Partner with engineers to implement and scale evaluation approaches in production systems.
• Work with AI labs and enterprise customers to understand evaluation needs and shape research priorities.
• Share research findings including through publications where appropriate.

• At least 3 years of full-time AI/ML research experience particularly in benchmarking evaluation methods NLP or language models.
• A record of published research with applied research experience and practical systems work valued.
• Strong Python skills and experience building clean maintainable software; familiarity with Django Flask or similar HTTP frameworks.
• Hands-on experience with PyTorch or TensorFlow language modeling LLM infrastructure and relevant AI/ML techniques.
• Experience creating benchmarks or evaluation methodologies and translating research into scalable systems.
• A masters or PhD in computer science machine learning or a related field is preferred; candidates with a bachelors degree and 1 to 3 years of applied experience may also be considered.
• Experience collaborating in engineering teams including Git workflows code review and incorporating feedback.

Base salary: $150000 to $200000 annually. Visa sponsorship is available.

On-site in San Francisco California United States.