Sr Applied Scientist, AI Agent Evaluation, AWS Applied AI Solution – Core Services

As part of the AWS Solutions organization we have a vision to provide business applications leveraging Amazons unique experience and expertise that are used by millions of companies worldwide to manage day-to-day operations. We will accomplish this by accelerating our customers businesses through delivery of intuitive and differentiated technology solutions that solve enduring business challenges. We blend vision with curiosity and Amazons real-world experience to build opinionated turnkey solutions. Where customers prefer to buy over build we become their trusted partner with solutions that are no-brainers to buy and easy to use.

Lead our pioneering AI initiative at AWS and define the future of AI agents. Shape how businesses leverage artificial intelligence by creating foundational building blocks used across AWS business applications.

The Core Services team within AWS Applied AI Solutions is creating the foundations that will power the next generation of AI agents from small business to enterprise scale. As a scientific leader youll drive our new agentic AI building blocks initiative pioneering the development of reusable AI components that accelerate and standardize AI product delivery across AWS business applications such as contact center supply chain healthcare and life sciences. Youll own one critical capability area such as agent identity and governance agent collaboration & orchestration agent evaluation or knowledge management and have the privilege to define what we build and how we build it.

This role combines the excitement of a startup environment with the scale of AWS. Youll research state-of-the-art open source and internal tools tackle highly ambiguous problems and lead scientific innovation building a V1 product from the ground up. You wont just be implementing someone elses vision youll chart the course define the roadmap and create novel scientific solutions that eliminate non-differentiating work while ensuring enterprise-grade quality and consistency. If you thrive on ownership are passionate about AI research and development and want to fundamentally influence how AWS builds AI products this role offers an extraordinary opportunity to make your mark.


Key job responsibilities
- Design and implement evaluation frameworks for AI agents including benchmarking tools annotation systems for RLHF and standardized patterns for memory orchestration and retrieval that ensure consistent performance across diverse use cases

- Develop deep expertise in a strategic research area within AI agent systems becoming the organizations scientific expert in areas such as agent design and evaluation.

- Identify and devise new research solutions for ill-defined customer or business problems that require novel methodologies and paradigms to be invented at the product level

- Articulate key potential scientific challenges of ongoing or future customer needs or business problems in AI agents and present interventions to address them

- Lead the design implementation and successful delivery of solutions for scientifically-complex problems writing critical-path code

- Write clear useful narratives and documentation describing inventions solutions and design choices that enable others to understand and reproduce your work

- Demonstrate detailed knowledge of your teams solutions and systems while proactively driving utilization and improvement upon the state of the art

- Independently assess alternative AI technologies and choose the right approaches to be integrated into your systems

- Apply and drive your team to adopt best practices in scientific research code development and technical documentation

- Influence across multiple teams to build consensus on scientific approaches and architectural decisions

- Actively mentor and develop other scientists and engineers elevating the technical capabilities of the organization

A day in the life
Your morning starts with a team standup followed by collaborative sessions with software engineers and product managers to understand requirements and internal customer needs. Youll dedicate focused time to researching the latest scientific advances in AI agents and designing novel approaches for your capability area. Throughout the day youll work with engineers and scientists within your team to implement solutions applying scientific rigor to validate approaches. You might lead a scientific design review gathering diverse perspectives to strengthen your methodologies. Your day could include conducting experiments analyzing results designing new evaluation methodologies meeting with internal customers to understand their AI agent needs or presenting your scientific vision to leadership stakeholders.

About the team
Applied AI Solutions Core Services is a comprehensive suite of core computing services and solutions provided by Amazon Web Services (AWS) to support the foundational infrastructure and operations needs of customers. It encompasses a wide range of essential services and capabilities across AWS. The Core Services team focuses on continuously improving the security availability and cost-effectiveness of these critical enterprise-wide services for AWS customers.

Why AWS
Amazon Web Services (AWS) is the worlds most comprehensive and broadly adopted cloud platform. We pioneered cloud computing and never stopped innovating thats why customers from the most successful startups to Global 500 companies trust our robust suite of products and services to power their businesses.
Diverse Experiences
Amazon values diverse experiences. Even if you do not meet all of the preferred qualifications and skills listed in the job description we encourage candidates to apply. If your career is just starting hasnt followed a traditional path or includes alternative experiences dont let it stop you from applying.
Work/Life Balance
We value work-life harmony. Achieving success at work should never come at the expense of sacrifices at home which is why we strive for flexibility as part of our working culture. When we feel supported in the workplace and at home theres nothing we cant achieve in the cloud.
Inclusive Team Culture
Here at AWS its in our nature to learn and be curious. Our employee-led affinity groups foster a culture of inclusion that empower us to be proud of our differences. Ongoing events and learning experiences including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences inspire us to never stop embracing our uniqueness.
Mentorship and Career Growth
Were continuously raising our performance bar as we strive to become Earths Best Employer. Thats why youll find endless knowledge-sharing mentorship and other career-advancing resources here to help you develop into a better-rounded professional.



- 3 years of building machine learning models for business application experience
- PhD or Masters degree and 6 years of applied research experience
- Experience programming in Java C Python or related language
- Experience with neural deep learning methods and machine learning
- Experience using managed ML/AI solutions
- Deep expertise in at least one AI/ML discipline such as NLP reinforcement learning etc.
- Experience with neural deep learning methods and popular frameworks such as PyTorch TensorFlow or MxNet
- Experience designing and implementing AI/ML systems including working with LLMs prompt engineering retrieval augmented generation (RAG) fine-tuning or AI agent development
- Experience using managed ML/AI solutions such as AWS SageMaker AI or Amazon Bedrock
- Demonstrated track record of scientific innovation with measurable business impact
- Experience building production systems that operate at scale
- Proven ability to drive scientific roadmap and secure management buy-in for new initiatives
- Strong collaboration skills with ability to influence across organizational boundaries

- Experience with modeling tools such as R scikit-learn Spark MLLib MxNet Tensorflow numpy scipy etc.
- Experience with large scale distributed systems such as Hadoop Spark etc.
- 7 years of experience applying scientific methods to solve complex AI problems at scale
- Experience mentoring junior scientists and engineers
- Advanced knowledge of large language models including fine-tuning evaluation and responsible AI practices
- Experience developing bias detection fairness metrics and ethical evaluation frameworks for AI systems
- Proven track record building reinforcement learning systems with human feedback including annotation frameworks and reward modeling
- Experience creating evaluation frameworks measuring factuality robustness and safety across diverse scenarios comparable to HELM or HealthBench
- Expertise in building reusable AI components with well-defined interfaces

Amazon is an equal opportunity employer and does not discriminate on the basis of protected veteran status disability or other legally protected status.

Our inclusive culture empowers Amazonians to deliver the best results for our customers. If you have a disability and need a workplace accommodation or adjustment during the application and hiring process including support for the interview or onboarding process please visit for more information. If the country/region youre applying in isnt listed please contact your Recruiting Partner.

Our compensation reflects the cost of labor across several US geographic markets. The base pay for this position ranges from $150400/year in our lowest geographic market up to $260000/year in our highest geographic market. Pay is based on a number of factors including market location and may vary depending on job-related knowledge skills and experience. Amazon is a total compensation company. Dependent on the position offered equity sign-on payments and other forms of compensation may be provided as part of a total compensation package in addition to a full range of medical financial and/or other benefits. For more information please visit This position will remain posted until filled. Applicants should apply via our internal or external career site.