Applied Scientist Conversational Assistant Services

As part of Alexa CAS team our mission is to provide scalable and reliable evaluation of the stateoftheart Conversational AI. We are looking for a passionate talented and resourceful Applied Scientist in the field of LLM Artificial Intelligence (AI) Natural Language Processing (NLP) to invent and build endtoend evaluation of how customers perceive stateoftheart contextaware conversational AI assistants.

A successful candidate will have strong machine learning background and a desire to push the envelope in one or more of the above areas. The ideal candidate would also have handson experiences in building Generative AI solutions with LLMs including Supervised FineTuning (SFT) InContext Learning (ICL) Learning from Human Feedback (LHF) etc.

As an Applied Scientist you will leverage your technical expertise and experience to collaborate with other talented applied scientists and engineers to research and develop novel methods for evaluating conversational assistants. You will analyze and understand user experiences by leveraging Amazons heterogeneous data sources and build evaluation models using machine learning methods.

Key job responsibilities
Design build test and release predictive ML models using LLMs
Ensure data quality throughout all stages of acquisition and processing including such areas as data sourcing/collection ground truth generation normalization and transformation.
Collaborate with colleagues from science engineering and business backgrounds.
Present proposals and results to partner teams in a clear manner backed by data and coupled with actionable conclusions
Work with engineers to develop efficient data querying and inference infrastructure for both offline and online use cases

About the team
Central Analytics and Research Science (CARS) is an analytics software and science team within Amazons Conversational Assistant Services (CAS) organization. Our mission is to provide an endtoend understanding of how customers perceive the assistants they interact with from the metrics themselves to software applications to deep dive on those metrics allowing assistant developers to improve their services. Learn more about Amazons approach to customerobsessed science on the Amazon Science website which features the latest news and research from scientists across the company. For the latest updates subscribe to the monthly newsletter and follow the @AmazonScience handle and #AmazonScience hashtag on LinkedIn Twitter Facebook Instagram and YouTube.

PhD or Masters degree and 4 years of CS CE ML or related field experience
Experience building machine learning models or developing algorithms for business application
Experience in patents or publications at toptier peerreviewed conferences or journals
Experience with programming languages such as Python Java C
Handson experience and deep understanding of the Large Language Models architectures

Research experience in conversational assistant or LLM evaluation
Publications at peerreviewed NLP/ML conferences (e.g. ACL EMNLP NAACL NeurIPS ICLR ICML AAAI etc.
Handson experience with using RLHF models

Amazon is committed to a diverse and inclusive workplace. Amazon is an equal opportunity employer and does not discriminate on the basis of race national origin gender gender identity sexual orientation protected veteran status disability age or other legally protected status.

Our inclusive culture empowers Amazonians to deliver the best results for our customers. If you have a disability and need a workplace accommodation or adjustment during the application and hiring process including support for the interview or onboarding process please visit for more information. If the country/region youre applying in isnt listed please contact your Recruiting Partner.

Our compensation reflects the cost of labor across several US geographic markets. The base pay for this position ranges from $136000/year in our lowest geographic market up to $223400/year in our highest geographic market. Pay is based on a number of factors including market location and may vary depending on jobrelated knowledge skills and experience. Amazon is a total compensation company. Dependent on the position offered equity signon payments and other forms of compensation may be provided as part of a total compensation package in addition to a full range of medical financial and/or other benefits. For more information please visit This position will remain posted until filled. Applicants should apply via our internal or external career site.