Sr. Applied Scientist, Alexa AI

As part of the Alexa AI team our mission is to provide scalable and reliable evaluation of state-of-the-art Conversational AI. We are looking for a passionate talented and resourceful Applied Scientist in the field of Large Language Models (LLMs) Artificial Intelligence (AI) and Natural Language Processing (NLP) to invent and build the end-to-end evaluation of how customers perceive state-of-the-art context-aware conversational AI assistants.

A successful candidate will have a strong machine learning background a deep understanding of the conversational AI stack and a desire to push the envelope in conversational-AI evaluation. As a senior member of the team you will own ambiguous high-impact evaluation problems end-to-end from defining the scientific direction to shipping the models and metrics that millions of customers and the developers who build for them depend on. The ideal candidate has hands-on experience building Generative AI solutions with LLMs including LLM-as-a-Judge (LLMaaJ) model distillation Supervised Fine-Tuning (SFT) In-Context Learning (ICL) and Learning from Human Feedback (LHF).

As an Applied Scientist you will leverage your technical expertise to set the research agenda for how we measure conversational quality mentor other scientists and engineers and partner across science engineering and business teams to research and develop novel evaluation methods. You will analyze and understand customer experiences using Amazons heterogeneous data sources and design train and maintain the evaluation models that serve as the source of truth for assistant quality.


Key job responsibilities
* Own the design development and long-term maintenance of flagship quality-evaluation metrics for a state-of-the-art conversational assistant spanning ground-truth definition data preparation model training and production maintenance.
* Research and build LLM-based evaluators including LLM-as-a-Judge systems and distill large judge models into efficient cost-effective models suitable for scaled online use.
* Set the technical direction for evaluation science and raise the bar for scientific rigor across the team; mentor scientists and engineers and review their work.
* Ensure data quality throughout all stages of acquisition and processing including data sourcing/collection ground-truth generation normalization and transformation.
* Present proposals and results to partner teams and leadership in a clear manner backed by data and coupled with actionable conclusions.
* Partner with engineers to develop efficient data-querying and inference infrastructure for both offline and online use cases.

About the team
Central Analytics and Research Science (CARS) is an analytics software and science team within Amazons Alexa AI organization. Our mission is to provide an end-to-end understanding of how customers perceive the assistants they interact with from the metrics themselves to software applications to deep dive on those metrics allowing assistant developers to improve their services. Learn more about Amazons approach to customer-obsessed science on the Amazon Science website which features the latest news and research from scientists across the company. For the latest updates subscribe to the monthly newsletter and follow the @AmazonScience handle and #AmazonScience hashtag on LinkedIn Twitter Facebook Instagram and YouTube.

- PhD or Masters degree and 5 years of applied research experience
- 3 years of building machine learning models for business application experience
- Experience programming in Java C Python or related language
- Experience with neural deep learning methods and machine learning
- Experience in developing and deploying LLMs in production on GPUs Neuron TPU or other AI acceleration hardware or experience building complex software systems that have been successfully delivered to customers

- Experience with Machine Learning and Large Language Model fundamentals including architecture training/inference lifecycles and optimization of model execution or experience with vLLM SGLang TensorRT or similar platforms in production environments
- Research or applied experience in conversational-assistant or LLM evaluation including ownership of a production quality metric.

Amazon is an equal opportunity employer and does not discriminate on the basis of protected veteran status disability or other legally protected status.

Los Angeles County applicants: Job duties for this position include: work safely and cooperatively with other employees supervisors and staff; adhere to standards of excellence despite stressful conditions; communicate effectively and respectfully with employees supervisors and staff to ensure exceptional customer service; and follow all federal state and local laws and Company policies. Criminal history may have a direct adverse and negative relationship with some of the material job duties of this position. These include the duties and responsibilities listed above as well as the abilities to adhere to company policies exercise sound judgment effectively manage stress and work safely and respectfully with others exhibit trustworthiness and professionalism and safeguard business operations and the Companys reputation. Pursuant to the Los Angeles County Fair Chance Ordinance we will consider for employment qualified applicants with arrest and conviction records.

Our inclusive culture empowers Amazonians to deliver the best results for our customers. If you have a disability and need a workplace accommodation or adjustment during the application and hiring process including support for the interview or onboarding process please visit for more information. If the country/region youre applying in isnt listed please contact your Recruiting Partner.

The base salary range for this position is listed below. Your Amazon package will include sign-on payments and restricted stock units (RSUs). Final compensation will be determined based on factors including experience qualifications and location. Amazon also offers comprehensive benefits including health insurance (medical dental vision prescription Basic Life & AD&D insurance and option for Supplemental life plans EAP Mental Health Support Medical Advice Line Flexible Spending Accounts Adoption and Surrogacy Reimbursement coverage) 401(k) matching paid time off and parental leave. Learn more about our benefits at WA Seattle - 167100.00 - 226100.00 USD annually