Applied Scientist, Customer Experience and Business Trends, Customer Experience and Business Trends

Amazons Customer Experience and Business Trends (CXBT) is looking for an Applied Scientist to help shape the science of evaluating agent-driven builder experiences on AWS. As builders increasingly delegate goals to autonomous coding agents the agents experience becomes the builders experience and measuring it rigorously is an open research problem. At Amazon Benchmarking we are developing new scientific methodologies to evaluate how autonomous agents perceive reason about and build on AWS.

This role sits at the intersection of GenAI agent evaluation and developer-experience research. You will design principled measurement methodologies that treat autonomous agents as instruments capturing behavioral evidence rather than self-reported outcomes and develop rigorous reproducible techniques for reasoning about why agents make the architectural decisions they do. Evaluating those decisions demands a scientist who understands cloud architecture and services in depth: to reason about the choices an agent makes you must understand the trade-offs it is navigating. Our teams mission is to constantly challenge the status quo and drive innovation to improve customer experience.

This role requires an expert in the areas of GenAI and machine learning with a keen interest in research. The ideal candidate will have experience with deep learning models for natural language processing tasks agentic/tool-using systems and the evaluation of autonomous agents along with a deep working knowledge of cloud architecture and services.

A successful candidate will have a passion for clarity be a self-starter comfortable with ambiguity have strong attention to detail be able to work in a fast-paced and entrepreneurial environment and be driven by a desire to innovate. Given the complex nature of the questions we are working on the ideal candidate will be relentlessly curious and be able to/be interested in having the business conversation underlying the data.

Key job responsibilities
- Develop novel scientific methodologies for evaluating autonomous tool-using agents advancing the state of the art in agent behavioral measurement and developer-experience research.
- Design rigorous reproducible experiments that yield causal insight into agent decision-making controlling for confounds and ensuring findings are grounded in observed behavior rather than self-report.
- Apply deep knowledge of cloud architecture and services to reason about the architectural decisions agents make the service trade-offs integration constraints and deployment considerations that shape what gets built.
- Develop verification and synthesis methods that ensure only evidence-grounded conclusions survive advancing techniques for reliability and faithfulness in agent evaluation.
- Design evaluation methodologies and metrics that quantify the quality of agent-produced outcomes across dimensions such as correctness robustness and security posture.
- Publish and present your work at internal and external scientific venues in the fields of ML.
- Collaborate with scientists engineers and product managers to design and implement science-focused systems.
- Proficiency in model development validation and implementation for NLP applications.
- Excellent verbal and written communication and presentation skills ability to convey rigorous mathematical concepts and considerations to non-experts.

A day in the life
As a scientist on the team you will be involved in every aspect of the process from idea generation business analysis and scientific research through development and deployment of advanced models and agentic systems giving you a real sense of ownership. Youll research design and build new methodologies to evaluate the experience of autonomous agents building on AWS developing the measurement science that lets us reason rigorously about what agents do and why. Besides theoretical analysis and innovation you will work closely with talented engineers and put your algorithms and models into practice.

About the team
Customer Experience and Business Trends (CXBT) is an organization made up of a diverse suite of functions dedicated to deeply understanding and improving customer experience globally. We are a team of builders that develop products services ideas and various ways of leveraging data to influence product and service offerings for almost every business at Amazon for every customer (e.g. consumers developers sellers/brands employees investors streamers gamers).

Our approach is based on determining the customer need along with problem solving and we work backwards from there. We use technical and non-technical approaches and stay aware of industry and business trends. We are a global team made up of a diverse set of profiles skills and backgrounds including: Scientists Product Managers Software Developers Computer Vision experts Solutions Architects Business Intelligence Engineers Business Analysts Risk Managers and more.

- 3 years of building models for business application experience
- PhD or Masters degree and 4 years of CS CE ML or related field experience
- Experience in patents or publications at top-tier peer-reviewed conferences or journals
- Experience programming in Java C Python or related language
- Experience in any of the following areas: algorithms and data structures parsing numerical optimization data mining parallel and distributed computing high-performance computing
- Hands-on experience (academic or industrial) building ML models for natural language processing tasks using GenAI

- Deep working knowledge of cloud architecture and services (e.g. compute storage databases auth networking) and the trade-offs involved in composing them into production systems
- Experience with agentic/tool-using systems LLM-based agents or the evaluation of autonomous agents
- Experience using cloud-based services and big data processing technologies such as AWS Redshift EMR Spark and Sagemaker
- Prior work experience as a research scientist or applied scientist at a consumer product company
- Excellent problem-solving ability. Demonstrable track record of dealing well with ambiguity prioritizing needs and delivering results in a dynamic environment
- Experience in professional software development

Amazon is an equal opportunity employer and does not discriminate on the basis of protected veteran status disability or other legally protected status.

Our inclusive culture empowers Amazonians to deliver the best results for our customers. If you have a disability and need a workplace accommodation or adjustment during the application and hiring process including support for the interview or onboarding process please visit for more information. If the country/region youre applying in isnt listed please contact your Recruiting Partner.

The base salary range for this position is listed below. Your Amazon package will include sign-on payments and restricted stock units (RSUs). Final compensation will be determined based on factors including experience qualifications and location. Amazon also offers comprehensive benefits including health insurance (medical dental vision prescription Basic Life & AD&D insurance and option for Supplemental life plans EAP Mental Health Support Medical Advice Line Flexible Spending Accounts Adoption and Surrogacy Reimbursement coverage) 401(k) matching paid time off and parental leave. Learn more about our benefits at WA Seattle - 142800.00 - 193200.00 USD annually