EY GDS Consulting AIA QE Engineer- Senior

Career Family AIA AI Evaluation & QE Engineer

Role Type Full Time

The opportunity

We are the only professional services organization who has a separate business dedicated exclusively to the financial services marketplace. Join the Digital Engineering Team and work with multidisciplinary teams from around the world to deliver a global perspective. Aligned to key industry groups including Asset Management Banking and Capital Markets Insurance and Private Equity Health Government Power and Utilities we provide integrated advisory assurance tax and transaction services. Through diverse experiences worldclass learning and individually tailored coaching you will experience continuous professional development. Thats how we develop outstanding leaders who deliver on our promises to clients communities and each other.

As the firm accelerates enterprisescale adoption of Generative AI we are establishing a formal AI Quality & Evaluation capability as a critical foundation for trustworthy productiongrade AI systems. This role plays a pivotal part in building that capability and ensuring AI systems are accurate grounded safe reproducible and compliant before reaching production.

We are looking for a Senior - AI Evaluation & QE Engineer (SOFTWARE ENGINEER) to join our AI Enabled Automation practice with 5 years of experience in Quality Engineering and Test Automation and strong hands on exposure to GenAI Large Language Models (LLMs) and RAG based systems.

This role requires deep expertise in Python based automation LLM evaluation AI system validation combined with strong engineering fundamentals to support enterprise AI project deployments across multiple engagements.

In this role you will contribute to two key capacities:

• Serving as the AI Evaluation & Quality Engineering Owner for GenAI and LLM-powered solutions with responsibilities including defining AI acceptance criteria benchmarking model behavior validating Retrieval-Augmented Generation (RAG) pipelines and implementing automated evaluation gates to ensure production readiness.
• Providing support for broader AI software engineering and deployment initiatives such as CI/CD integration cloud-native deployments backend validation and comprehensive quality engineering across enterprise AI platforms and use cases.

EYs Artificial Intelligence & Automation (AIA) team is a specialized intelligence driven business unit that combines advanced AI automation engineering and deep domain knowledge to deliver transformative enterprise solutions. The AIA practice partners with clients to analyze design and operationalize AI led automation strategies that accelerate digital transformation and unlock measurable business value. We address organizations most critical challenges across process optimization intelligent workflows data driven decisioning Generative AI adoption and automation-enabled innovation. With a unique ability to translate AI strategy into actionable architectures scalable platforms and production grade automations EYs AIA team helps clients build sustainable competitive advantage enhance productivity and navigate disruption by embedding intelligence at the core of their business and technology ecosystems.

Your key responsibilities

AI Strategy & Solution Design

• Act as the AI Evaluation & Quality engineering owner within AI Factory / POD based delivery models
• Partner with AI Engineers backend teams product owners and architects to define AI acceptance criteria and quality gates
• Support end to end validation of GenAI application architectures including API layers orchestration logic and backend services
• Evaluate AI readiness across environments (DEV / QA / PROD) and ensure quality consistency across deployments
• Contribute to enterprise AI solution design discussions with a focus on trust reliability

Responsible AI

Collaborate closely with product UX and platform teams to align AI quality with functional and business goals

GenAI & Agentic AI Development

• Evaluate and benchmark AI outputs across multiple LLMs (model agnostic) including GPT Claude LLaMA Gemini and enterprise models
• Validate Retrieval Augmented Generation (RAG) pipelines end to end including retrieval accuracy grounding chunking strategies and edge cases
• Implement automated AI evaluation pipelines using DeepEval and RAGAs to assess:
  • Accuracy and correctness
  • Faithfulness and hallucination risk
  • Relevance and contextual grounding
  • Consistency and reproducibility
  • Safety toxicity and policy compliance
• Develop LLM regression test suites covering prompt changes model upgrades orchestration logic and workflow variations
• Support synthetic data generation and adversarial testing to stress test AI behavior under non deterministic conditions
• Apply structured analytical thinking to reason about ambiguous and probabilistic AI outputs

MLOps Cloud & Observability

• Build and maintain scalable modular AI test automation frameworks using Python and PyTest
• Integrate AI evaluation pipelines into CI/CD workflows (Azure DevOps GitHub Actions)
• Establish quality baselines benchmarks and regression thresholds for AI systems prior to production rollout
• Support deployment validation for AI workloads on Azure AWS or GCP
• Contribute to AI observability by validating evaluation metrics monitoring signals and responsible AI guardrails
• Ensure AI systems meet enterprise standards for reliability traceability and reproducibility

Team Leadership & Collaboration

• Mentor QE and AI engineers on GenAI quality engineering best practices
• Drive quality awareness across engineering teams shifting AI validation from implicit to governed
• Participate in sprint planning testing cycles release sign off and hypercare for AI deployments
• Communicate evaluation findings risks and recommendations clearly to both technical and business stakeholders
• Stay current with advancements in GenAI evaluation frameworks LLM testing patterns and Responsible AI practices

Required Skills

• 57 years of hands-on experience in AI Evaluation Engineering combining expertise in Quality Engineering and Test Automation for AI and GenAI systems
• Strong proficiency in Python with deep experience building PyTest based automation frameworks
• Hands on experience with DeepEval and RAGAs for LLM and RAG evaluation
• Solid understanding of LLM behavior prompt engineering concepts and RAG architectures
• Experience testing web based AI applications APIs and backend services
• Strong analytical skills to reason about non deterministic and probabilistic AI outputs
• Working knowledge of cloud platforms (Azure / AWS / GCP) and CI/CD pipelines Azure exp is must.

Good to Have

• Backend development exposure (FastAPI Flask RESTful services)
• Experience integrating QE pipelines with Azure DevOps
• Familiarity with LLMOps / MLOps concepts such as model versioning and rollout validation
• Exposure to Responsible AI frameworks safety testing and governance controls
• Experience working in AI Factory POD based or agile delivery models
• Exposure to DevOps tools such as Terraform GitHub Actions Docker

Preferred Qualifications

• Prior experience supporting multiple enterprise AI project deployments across domains with Azure AI Stack Foundry is plus.
• Certifications in Azure AI additional AWS Machine Learning or GCP Professional ML Engineer is plus
• Certifications or hands on experience in MLOps Kubernetes or DevOps for AI

Education

Degree: Bachelors or Masters degree in computer science Artificial Intelligence Data Science Engineering Mathematics or a related field or equivalent practical experience

What we offer

EY Global Delivery Services (GDS) is a dynamic and truly global delivery network. We work across six locations Argentina China India the Philippines Poland and the UK and with teams from all EY service lines geographies and sectors playing a vital role in the delivery of the EY growth strategy. From accountants to coders to advisory consultants we offer a wide variety of fulfilling career opportunities that span all business GDS you will collaborate with EY teams on exciting projects and work with well-known brands from across the globe. Well introduce you to an ever-expanding ecosystem of people learning skills and insights that will stay with you throughout your career.

• Continuous learning: Youll develop the mindset and skills to navigate whatever comes next.
• Success as defined by you: Well provide the tools and flexibility so you can make a meaningful impact your way.
• Transformative leadership: Well give you the insights coaching and confidence to be the leader the world needs.
• Diverse and inclusive culture: Youll be embraced for who you are and empowered to use your voice to help others find theirs.