LLM Quality Engineer

In AECOMs AI Engineering team your code will directly shape the physical world around us. We build AI-driven technology that revolutionises how infrastructure and buildings are designed and engineered; reducing waste cutting CO₂ and making the built environment more efficient and sustainable. This is where software has measurable real-world impact. 

With our AI Engineering team weve created a unique setup: a lean highly technical team with the speed and ownership of a start up backed by the scale resources and domain expertise of one of the worlds leading engineering firms. 

There has never been a better time to be at AECOM. We are leading the industrys AI transformation and with our people and technology we deliver excellence and innovate with impact. 

We invite you to bring your bold ideas and big dreams to solve the worlds most complex challenges. Were one global team driven by our common purpose to deliver a better world. Join us. 

What Youll Do 

As part of the Assurance & Insight team you will focus on ensuring the quality and reliability of our AI-driven systems. This role sits at the intersection of testing AI evaluation and operational quality.  

You will help build the foundations for how we test benchmark and monitor both software systems and LLM-driven outputs ensuring that what we deploy performs as expected in real-world scenarios. 

You will take ownership of quality across our AI systems from automated testing to LLM evaluation and production monitoring. Your work will help reduce regressions improve reliability and free up engineering time to focus on building. 

Were not looking for years of experience. Were looking for someone detail-oriented who wants to own a domain can write clean Python and is genuinely curious about how AI systems behave. Its a good fit for a recent graduate who wants end-to-end ownership early in their career is comfortable working close to AI systems and cares about making quality measurable rather than subjective. 

Qualifications :

Must-Have Qualifications 

• Degree in computer science software engineering or a related technical field or equivalent demonstrable experience 
• Exposure to LLMs or AI systems (even personal projects or coursework counts)  
• Familiarity with test frameworks such as pytest or similar  
• Understanding of CI/CD concepts and release pipelines  
• Comfortable working with Python and APIs  
• Experience writing or evaluating prompts for language models  
• Strong attention to detail and a methodical approach to problem-solving 

Preferred Skills 

• Exposure to AI/LLM tools or projects  
• Experience with testing QA or automation (academic or personal projects)  
• Interest in data quality monitoring or analytics  
• Experience working in team-based or project-based environments 

Ytterligere informasjon :

Our Hiring Process 

• 25-minute screening call 
• Take-home challenge: A hands-on task to assess your problem-solving and technical skills 
• Combined technical and cultural interview (in-person) 
  • Whiteboard Interview: 1-hour with 2 of our engineers to discuss your solution to the take-home challenge 
  • Culture fit: 30-minute meeting with our leadership team 

Why Join Us 

• Work on real-world problems where AI creates measurable impact. 
• Be part of a team where your work matters and your ideas become real. 
• Collaborate with sharp driven colleagues in a culture of trust ownership and high standards. 
• Contribute to making the built environment smarter and more sustainable. 

Remote Work :

No

Employment Type :

Full-time