QA Engineer AI Systems (Senior Consultant) AI Hub Amman

Within Assurance we assist our clients to address the accounting and financial reporting challenges facing their business. You will be part of a team that provides insight and services that accelerate analytics decision-making and innovation to build a stronger and more efficient finance function. You will experience ongoing professional development through diverse experiences world-class learning and individually tailored coaching.

That is how we develop outstanding leaders who team to deliver on our promises to all of our stakeholders and in so doing play a critical role in building a better working world for our people for our clients and for our communities.

Sound interesting Well this is just the beginning. Because whenever you join however long you stay the exceptional EY experience lasts a lifetime.

The opportunity

As a Senior QA Engineer - AI Systems in the AI Hub you will test evaluate and improve AI products before they reach clients and production. You willvalidateend-to-end AI solutions (LLMs agents RAG OCR/document intelligence traditional ML prompts/tools and data pipelines) and the human workflows around them.

Your key responsibilities

You will partner with AI engineers full stack/data/DevOps teams product and security to define quality and risk build evaluation approaches surface failure modes and ensure solutions are reliable safe explainable andenterprise-ready.

Skills and attributes for success

To qualify for the role you must have

• • 7 years in QA engineering test automation AI/ML evaluation data quality or related roles

  • Hands-on testing of AI/ML/GenAI/OCR/data solutions beyond traditional UI/API checks

  • Working knowledge of LLMs prompts embeddings/vector search RAG agents OCR/document extraction and model-serving APIs

  • Test strategy for probabilistic/context-dependent systems and non-deterministic outputs

  • Build and maintain eval datasets (golden sets adversarial/edge cases) and regression suites

  • Evaluate quality with fit-for-purpose metrics ( task success hallucinations OCR accuracy precision/recall latency cost)

  • Experience with test automation using Python TypeScript/JavaScript or similar scripting languages

  • Familiarity with Azure AI/Azure OpenAI/Azure AI Foundry/Document Intelligence or similar platforms

  • Understanding of responsible AI and enterprise risk (safety privacy bias/fairness explainability)

  • Experience in iterative delivery teams (PoCs/MVPs to production)

Ideally youll also have

• • Experience with AI evaluation tools or frameworks such as Azure AI Foundry evaluatorspromptfoo RagasDeepEvalMLflowLangSmith Weights & Biases or similar tools

  • Experience testing agentic systems built with frameworks such asLangChainLangGraph Semantic KernelAutoGen OpenAI Agents SDK or Microsoft Agent Framework

  • Experience with LLM-as-judge evaluation patterns custom evaluators human-in-the-loop review workflows and evaluation rubric design

  • Experience with document-heavy AI systems OCR quality assessment Arabic/English content financial/regulatory documents or enterprise knowledge bases

  • Experience with responsible AI AI governance NIST AI RMF OWASP Top 10 for LLM Applications OWASP Machine Learning Security Top 10 or similar AI risk frameworks

  • Experience with cybersecurity or AI red-teaming techniques especially prompt injection data leakage excessive agency unsafe tool use vector/embedding weaknesses and model manipulation risks

  • Experience with Azure Monitor Application InsightsOpenTelemetrytraces or production AI observability dashboards

  • Experience with Playwright Cypress PostmanpytestVitest or other automation tools used to support end-to-end quality gates

  • Experience supporting regulated or security-sensitive environments where privacy auditability access control and data residency matter

  • GenAI engineering exposure (prompting RAG design model selection eval harnesses and secure deployment patterns)

What we look for

We are interested in entrepreneurs who have the confidence to develop and promote a brand new strategic vision both internally and externally. You will be business savvy with a passion for innovation as well as the motivation to create your own EY journey.

What we offer

We offer a competitive compensation package where youll be rewarded based on performance and recognized for the value you bring to our business. Plus we offer:

• Continuous learning: Youll develop the mindset and skills to navigate whatever comes next.
• Success as defined by you: Well provide the tools and flexibility so you can make a meaningful impact your way.
• Transformative leadership: Well give you the insights coaching and confidence to be the leader the world needs.
• Diverse and inclusive culture: Youll be embraced for who you are and empowered to use your voice to help others find theirs.

If you can demonstrate that you meet the criteria above please contact us as soon as possible.

The exceptional EY experience. Its yours to build.

EY Building a better working world

EY exists to build a better working world helping to create long-term value for clients people and society and build trust in the capital markets.

Enabled by data and technology diverse EY teams in over 150 countries provide trust through assurance and help clients grow transform and operate.

Working across assurance consulting law strategy tax and transactions EY teams ask better questions to find new answers for the complex issues facing our world today.