Director AI Systems Reliability, Testing & Performance

Salary Range:

100240.00 - 186160.00

Job Description Summary

Director: AI Systems Reliability Testing & Performance

#LI-Hybrid

Location: London
Novartis is unable to offer relocation support for this role: please only apply if this location is accessible for you.

The Director AI Systems Reliability Testing & Performance is a senior technical AI leadership role within Data Science & AI responsible for defining how AI systems across Novartis Development are evaluated tested validated benchmarked monitored and continuously improved throughout their lifecycle.
This role owns the technical evidence required to determine whether AI systems are reliable robust secure and fit-for-use across Novartis Development. The role defines common approaches for AI evaluation benchmarking testing validation monitoring and production-readiness across agentic AI systems predictive models retrieval systems digital twins and other AI capabilities.
The Director provides technical leadership in AI evaluation science reliability engineering validation adversarial testing and performance assessment. Key areas of focus include model and agent evaluation benchmarking failure-mode analysis drift detection digital twin validation AI red teaming observability traceability and technical evidence generation supporting regulated and business-critical AI systems.
This is not a Governance Product Management PMO or infrastructure operations role. Governance owns policies risk frameworks and approval processes. Product teams own roadmaps adoption and value realization. DDIT and engineering teams own platforms infrastructure and operational services.
Success means Development AI systems are supported by objective evidence demonstrating how they perform where they fail and whether they remain fit-for-use over time.

Job Description

Major Accountabilities

AI Evaluation & Benchmarking

• Define evaluation methodologies for AI systems models agents digital twins and simulation environments across Development.
• Establish benchmark suites evaluation datasets and testing harnesses used across AI initiatives.
• Define objective measures for quality reliability robustness grounding agent effectiveness and task success.
• Ensure evaluation approaches remain scientifically rigorous reproducible and comparable across AI systems.
• Build a common evidence framework for assessing AI capabilities limitations and fitness-for-use.

AI Testing & Validation

• Establish approaches for hallucination testing failure-mode analysis robustness testing and behavioral validation.
• Define validation methodologies for agentic systems digital twins simulation environments and human-in-the-loop workflows.
• Develop production readiness criteria for AI systems operating in Development environments.
• Support technical validation activities required for GxP-relevant and regulated AI systems.
• Ensure AI systems are tested under realistic operating conditions and supported by objective validation evidence.

Reliability Monitoring & Performance

• Define how AI system reliability degradation drift and operational performance are measured over time.
• Establish monitoring requirements and performance assessment frameworks across Development AI systems.
• Partner with engineering teams to ensure required evaluation monitoring tracing and observability signals are available.
• Define methods for identifying diagnosing and assessing AI system failures.
• Promote evidence-based improvement of deployed AI capabilities.

AI Security & Adversarial Testing

• Define approaches for AI red teaming adversarial testing and security evaluation.
• Establish testing methodologies for prompt injection jailbreaks retrieval attacks tool misuse and agent manipulation scenarios.
• Assess AI-specific vulnerabilities and resilience risks in collaboration with cybersecurity teams.
• Ensure security testing forms part of AI system validation and production-readiness assessments.

Technical Evidence & AI Performance Intelligence

• Establish portfolio-wide approaches for capturing technical evidence related to AI quality reliability robustness and performance.
• Define standard reporting for benchmark results validation findings monitoring signals and reliability assessments.
• Create transparency into AI system strengths limitations and failure modes.
• Generate technical evidence supporting validation audit inspection and governance activities.
• Enable objective decisions on AI system readiness reliability and fitness-for-use.

Key Performance Indicators

Percentage of AI systems meeting assurance standards.

Coverage of evaluation and monitoring across AI portfolio.

Reliability and availability of business-critical AI solutions.

Audit and compliance readiness.

Time to identify and resolve AI performance issues.

Percentage of AI initiatives with measurable business outcomes.

Leadership confidence in AI performance reporting.

Minimum Requirement: Work Experience

• 10 years in AI machine learning software engineering quality risk validation or technology governance.
• Experience deploying or overseeing business-critical AI systems.
• Experience establishing governance controls quality standards or operational frameworks.
• Experience defining evaluation approaches and performance metrics.
• Experience managing operational risk in regulated environments.
• Experience partnering with infrastructure and platform teams.

ewards

At Novartiswerecommitted to reimagining medicine together - and rewarding the people who make it happen.

The rewards of being part of our team go far beyond base pay and also offer a variety of competitive benefits inkindto help you thrive personally and professionally such as insurance plans retirement plans wellbeingresourcesand global recognition addition we provide flexible and hybrid working options where possible anda minimum of14 weekspaid parental leave.

Expected Annual Base Salary Range for role:

• UK: 100240.00 - 186160.00 GBP Annual

The salary offered isdeterminedbased on gender-neutralobjectives such as relevant skillscompetenciesand experiencein accordance withthe Novartis pay setting policy and upon joining Novartis will be reviewed periodically.

In addition to your base salary you may be eligible for a performance-based bonus depending on certain performance parameters. Further details will be provided during the application process.

Pay equity is a fundamental principle of our employment policy and reflects our commitment to create a diverseequitableand inclusive environment that treats all employees with dignity and respect as outlined in our Code of Ethics.

Readourbrochureto learn more about our global total rewards offering: Benefits and compensation may vary by country and are subject to local legal requirements including provisions of collective bargaining agreements where applicable. A full overview of your compensation package including any relevant collective bargaining agreement details applicable to your role based on your employment location and Novartis employer entity will be communicated separately to you during the applicationprocess.

Commitment to Diversity and Inclusion / EEO paragraph:

Novartis is committed to building an outstanding inclusive work environment and diverseteamsrepresentativeof the patients and communities we serve.

Why Novartis:Helping people with disease and their families takes more than innovative science. It takes a community of smart passionate people like you. Collaboratingsupportingand inspiring each other. Combining to achieve breakthroughs that change patients lives. Ready to create a brighter future together Desired

Artificial Intelligence (AI) Business Value Creation Change Management Curious Mindset Data Governance Data Literacy Data Quality Data Science Data Visualization Deep Learning Learning Agility Machine Learning (ML) Machine Learning Algorithms Mentorship Stakeholder Engagement Statistical Analysis Time Series Analysis