Agentic Evaluation Scientist, Amazon Customer Service

Amazons Customer Service (CS) is undergoing a major transformation leveraging agentic AI techniques to investigate why customers experience defects and how to prevent them in the future. Were looking for scientists to help drive the quality and capabilities of these agentic systems.

Amazons Customer Experience Improvement (CXI Tech) team is looking for a scientists to help drive rigorous development in the agentic era. We seek to identify and eliminate unsatisfactory purchasing experiences that may cause customers to return items or not return to Amazon. This role requires you to re-imagine the way we leverage customer interactions/feedback for defect identification and the actions we take for reliable downstream defect resolution for an improved customer experience throughout the consumer business. Driven by the vision that no defect need ever impact customers twice we work to develop rigorously tested AI systems to investigate what caused individual defects experienced by customers and identify the needed changes to ensure they do not occur particular we are seeking scientists with a proven track record of making efficient use of human annotations to evaluate AI systems.

Come help Amazon ensure the best possible experience for our customers!

Key job responsibilities
- Develop evaluation techniques for agentic solutions that operate at Amazon scale developing techniques to make efficient use of human annotations to establish rigorous performance of agentic systems.
- Develop tools and patterns that improve the performance of agentic investigation systems.
- Create standalone ML powered models when needed to support automated investigation of customer defects.

A day in the life
- Develop techniques to measure and improve the performance of agentic systems even when the performance of those systems exceed the existing human benchmark
- Interface closely with Engineering and Product partners to align the direction of scientific development with needs of the greater business
- Publish techniques internally and externally (when possible) to promote greater rigor in agentic development

About the team
CXI Tech team owns building technology products to drive defect elimination across Amazon and at scale to safeguard and improve the customer experience across all interactions with Amazon.

- 3 years of building models for business application experience
- PhD or Masters degree and 4 years of CS CE ML or related field experience
- Experience in patents or publications at top-tier peer-reviewed conferences or journals
- Experience programming in Java C Python or related language
- Experience in any of the following areas: algorithms and data structures parsing numerical optimization data mining parallel and distributed computing high-performance computing

- Experience using Unix/Linux
- Experience in professional software development
- Experience building machine learning models or developing algorithms for business application

Amazon is an equal opportunity employer and does not discriminate on the basis of protected veteran status disability or other legally protected status.

Our inclusive culture empowers Amazonians to deliver the best results for our customers. If you have a disability and need a workplace accommodation or adjustment during the application and hiring process including support for the interview or onboarding process please visit for more information. If the country/region youre applying in isnt listed please contact your Recruiting Partner.

The base salary range for this position is listed below. Your Amazon package will include sign-on payments and restricted stock units (RSUs). Final compensation will be determined based on factors including experience qualifications and location. Amazon also offers comprehensive benefits including health insurance (medical dental vision prescription Basic Life & AD&D insurance and option for Supplemental life plans EAP Mental Health Support Medical Advice Line Flexible Spending Accounts Adoption and Surrogacy Reimbursement coverage) 401(k) matching paid time off and parental leave. Learn more about our benefits at WA Seattle - 142800.00 - 193200.00 USD annually