Software QA Lead – AI & GenAI Platforms

Build the quality engine behind enterprise AI.

Were looking for a hands-on QA leader to drive the strategy and execution of quality assurance for enterprise-grade Generative AI assistant applications.

Youll own how AI systems are tested validated and scaled spanning full-stack applications powered by LLMs and RAG workflows. This role is critical to ensuring reliable AI outputs strong user experiences and responsible ethical AI behavior in production.

This position will offer flexibility for hybrid work schedules to include both in-office presence and telecommute/virtual work to be based in Chicago IL.

Key Responsibilities

• Define and execute QA strategy for end-to-end AI applications (frontend backend and AI workflows)

• Lead validation of LLM outputs for accuracy coherence and reliability across diverse scenarios

• Test and optimize RAG pipelines to ensure high-quality retrieval and grounded responses

• Design user-centric test scenarios (multi-turn conversations edge cases failure handling)

• Conduct performance load and latency testing across AI and application layers

• Build and scale automated testing frameworks (API UI integration) within CI/CD pipelines

• Establish and track quality metrics (accuracy relevance latency user satisfaction)

• Ensure ethical AI standards including bias detection and responsible output validation

• Partner cross-functionally to embed QA practices across the development lifecycle

Qualifications :

Minimum Qualifications

• Bachelors Degree plus at least 10 years of software development experience or demonstrated equivalency of experience and/or education

• Proven experience leading QA for complex full-stack applications

• Strong expertise in testing Generative AI systems (LLMs RAG workflows)

• Hands-on experience with test automation tools (e.g. Selenium Appium)

• Experience with API testing distributed systems and CI/CD pipelines

• Familiarity with performance testing tools (e.g. JMeter LoadRunner)

• Strong analytical problem-solving and communication skills

Preferred Qualifications

• Experience with AI evaluation techniques (prompt testing output scoring bias testing)

• Background in conversational AI NLP or chatbot testing

• Familiarity with enterprise data/retrieval systems and knowledge bases

• Experience working in cloud environments (AWS Azure or GCP)

Additional Information :

Additional Information

• Relocation assistance is not available for this position

• Sponsorship for US work authorization is not available for this position now or in the future.

About AECOM 

AECOM is proud to offer comprehensive benefits to meet the diverse needs of our employees. Depending on your employment status AECOM benefits may include medical dental vision life AD&D disability benefits paid time off leaves of absences voluntary benefits perks flexible work options well-being resources employee assistance program business travel insurance service recognition awards retirement savings plan and employee stock purchase plan. 

AECOM is the global infrastructure leader committed to delivering a better world. As a trusted professional services firm powered by deep technical abilities we solve our clients complex challenges in water environment energy transportation and buildings. Our teams partner with public- and private-sector clients to create innovative sustainable and resilient solutions throughout the project lifecycle from advisory planning design and engineering to program and construction management. AECOM is a Fortune 500 firm that had revenue of $16.1 billion in fiscal year 2025. Learn more at . 

What makes AECOM a great place to work 

You will be part of a global team that champions your growth and career ambitions. Work on groundbreaking projects - both in your local community and on a global scale - that are transforming our industry and shaping the future. With cutting-edge technology and a network of experts youll have the resources to make a real impact. Our award-winning training and development programs are designed to expand your technical expertise and leadership skills helping you build the career youve always envisioned. Here youll find a welcoming workplace built on respect collaboration and communitywhere you have the freedom to grow in a world of opportunity. 

As an Equal Opportunity Employer we believe in your potential and are here to help you achieve it. All your information will be kept confidential according to EEO guidelines. 

Remote Work :

No

Employment Type :

Full-time