Lead Responsible AI Engineer

ROLE SUMMARY

As a Lead Responsible AI (RAI) Engineer you will lead the design implementation and operationalization of responsible AI safety quality evaluation and governance practices across GenAI and agentic AI solutions. This role sits at the intersection of AI engineering quality engineering governance and production operations ensuring that AI systems are not only functional but also safe reliable auditable and fit for enterprise use.

You will work closely with AI engineers architects Team Leads product owners integration teams and platform teams to embed responsible AI controls into the full product lifecycle. This role requires strong understanding of LLM systems RAG pipelines agentic workflows prompt safety guardrails testing strategies observability and enterprise risk considerations. It is not a purely policy or compliance role it is a hands-on technical leadership role that ensures AI systems can be trusted in production.

The ideal candidate will combine technical depth in AI-enabled systems practical experience with testing and quality frameworks and the ability to guide teams toward safe scalable and supportable implementation patterns.

KEY RESPONSIBILITIES

• Lead the design and implementation of responsible AI safety and quality engineering practices for GenAI and agentic AI products
• Define and operationalize AI validation strategies covering functional correctness factual reliability hallucination risk retrieval quality prompt safety agent behavior tool use and failure handling
• Establish test approaches for LLM-powered applications RAG systems agentic workflows multi-step reasoning tool orchestration and context-sensitive AI interactions
• Design and implement evaluation frameworks for relevance safety groundedness consistency latency token usage and business outcome alignment
• Work with AI engineers and architects to embed guardrails prompt controls model usage boundaries escalation paths fallback strategies and safety-oriented design patterns
• Drive adoption of governance and auditability practices across AI solution development including development of automated test scripts enable traceability review checkpoints risk controls and evidence collection
• Partner with engineering and platform teams to implement AI observability logging trace analysis usage monitoring cost awareness and incident diagnostics
• Review solution designs to ensure they account for responsible AI data sensitivity model behavior risks compliance expectations and operational resilience
• Guide teams on how to test and validate agentic AI systems including state management tool-calling reliability context integrity multi-agent coordination and autonomous decision boundaries
• Contribute to the definition of engineering standards for prompt lifecycle management evaluation automation red-teaming adversarial testing and regression prevention
• Collaborate with product process and engineering stakeholders to balance innovation speed with risk management trust and enterprise readiness
• Mentor engineers and quality professionals on AI testing safety validation responsible AI engineering practices and production monitoring approaches
• Contribute reusable assets such as test harnesses prompt evaluation templates governance checklists safety review frameworks red-team patterns and validation accelerators

Required Qualifications

• 8 years of experience in software engineering AI engineering quality engineering test engineering AI governance or related technical roles including strong experience working with AI-enabled or ML-driven systems
• Proven experience designing or implementing quality safety validation governance risk compliance and auditability practices for AI-enabled systems in enterprise or production environments
• Strong understanding of GenAI and agentic AI systems including LLM workflows retrieval-augmented generation (RAG) prompt engineering model / tool interaction context-aware behavior and the failure modes associated with autonomous or semi-autonomous AI systems
• Practical knowledge of agentic AI patterns including orchestration workflows tool-calling behavior context management memory / state handling and multi-step AI interactions
• Experience defining and operationalizing test and evaluation strategies for AI systems including functional testing non-functional testing hallucination analysis retrieval validation regression testing groundedness evaluation red-teaming and scenario-based validation
• Strong understanding of responsible AI concerns including model safety fairness explainability transparency content controls enterprise risk management data sensitivity human oversight and governance-by-design
• Hands-on familiarity with enterprise responsible AI and governance frameworks such as NIST AI RMF and related GenAI / AI risk management approaches and the ability to translate such frameworks into practical engineering controls review criteria and release-readiness expectations
• Practical experience with fairness and bias evaluation tooling such as Fairlearn AIF360 or equivalent approaches used to assess fairness bias or model behavior in enterprise environments
• Hands-on experience with LLM evaluation tracing and quality analysis tools such as LangSmith TruLens DeepEval or equivalent frameworks used for prompt evaluation trace analysis groundedness checks workflow quality review and regression monitoring
• Familiarity with runtime safeguard and guardrail tooling such as Guardrails AI NeMo Guardrails or equivalent approaches used to enforce policy constrain unsafe behavior validate outputs and improve runtime reliability of GenAI or agentic AI systems
• Practical experience with enterprise AI governance monitoring and deployment tooling such as Azure AI Content Safety Azure Monitor Application Insights OpenTelemetry MLflow or equivalent platforms used for lifecycle tracking observability reproducibility monitoring and enterprise governance of AI systems
• Familiarity with broader enterprise AI governance and model oversight platforms such as Vertex AI monitoring or equivalent governance and monitoring ecosystems especially where multiple model policy and auditability controls are required
• Experience working with test automation validation frameworks CI/CD-driven quality controls and engineering delivery tooling such as Azure DevOps GitHub Git Python pytest API testing tools automated release controls and scenario-based validation approaches (or equivalent tools and frameworks)
• Ability to work closely with AI engineers architects Team Leads and product teams to convert responsible AI principles into implementable engineering controls validation strategies safety checks and production monitoring practices
• Strong SDLC ownership mindset including testing strategy release quality production monitoring governance checkpoints incident learning audit readiness and continuous improvement
• Strong communication and stakeholder alignment skills including the ability to discuss risk quality safety governance and production-readiness topics with both technical and non-technical stakeholders

Preferred Qualifications

• Experience testing or governing agentic AI workflows involving multi-step reasoning tool orchestration memory/context management or multi-agent coordination
• Exposure to Model Context Protocol (MCP) agent-to-agent (A2A) interaction patterns or equivalent context-sharing and distributed agent communication approaches
• Experience working with RAG evaluation retrieval quality measurement grounding validation chunking / retrieval strategy testing or vector search quality assessment
• Familiarity with Azure AI Search Pinecone Weaviate FAISS or equivalent retrieval and vector-enabled platforms
• Experience contributing reusable safety controls evaluation accelerators governance playbooks review templates or AI quality engineering frameworks
• Familiarity with token usage analytics cost controls prompt safety checks output filtering and human-in-the-loop escalation models
• Experience supporting technical direction for engineering teams through quality reviews design reviews validation sign-offs and mentoring
• Experience working in a build-own-operate product model where long-term supportability reliability and controlled evolution of AI systems are critical
• Ability to communicate technical risks and responsible AI decisions clearly to engineering leaders product stakeholders and governance team