Associate Director, Generative AI Evaluation & Standards

At Johnson & Johnsonwe believe health is everything. Our strength in healthcare innovation empowers us to build aworld where complex diseases are prevented treated and curedwhere treatments are smarter and less invasive andsolutions are our expertise in Innovative Medicine and MedTech we are uniquely positioned to innovate across the full spectrum of healthcare solutions today to deliver the breakthroughs of tomorrow and profoundly impact health for more at .

As guided by Our Credo Johnson & Johnson is responsible to our employees who work with us throughout the world. We provide an inclusive work environment where each person is considered as an individual. At Johnson & Johnson we respect the diversity and dignity of our employees and recognize their merit.

Job Function:

Job Sub Function:

Job Category:

All Job Posting Locations:

Job Description:

Associate Director Generative AI Evaluation & Standards

Location: This position will be located in either Barcelona or Madrid.

About J&J

At Johnson & Johnson we believe health is everything. Our strength in healthcare innovation empowers us to build a world where complex diseases are prevented treated and cured where treatments are smarter and less invasive and solutions are personal. Through our expertise in Innovative Medicine and MedTech we are uniquely positioned to innovate across the full spectrum of healthcare solutions today to deliver the breakthroughs of tomorrow and profoundly impact health for humanity.

Our expertise in Innovative Medicine is informed and inspired by patients whose insights fuel our science-based advancements. Visionaries like you work on teams that save lives by developing the medicines of tomorrow. Join us in developing treatments finding cures and pioneering the path from lab to life while championing patients every step of the way.

About the Role

Johnson & Johnson Innovative Medicine is recruiting an Associate Director GenAI Evaluation & Standards to join our Data Science and Digital Health team (DSDH). This is a newly created leadership role within the Generative AI organization reporting directly to the Head of Generative AI.

The GenAI team is deploying AI across discovery development regulatory and operations to create AI assistants and agentic systems across the therapeutic areas and functions. As we scale from proof-of-concept to enterprise deployment we need a dedicated evaluation function that defines what quality means for generative AI in a regulated pharmaceutical R&D environment communicates the metrics effectively and enforces that standard across every platform vendor and capability we build or adopt. This role owns evaluation and governance for all generative AI work across R&D.

Key Responsibilities

Evaluation Framework Design

• Design and maintain evaluation frameworks for GenAI solutions spanning LLM quality RAG performance agent reliability safety and scientific accuracy.

• Develop therapeutic-area-specific evaluation criteria with business teams reflecting domain-specific quality requirements per therapeutic area and application type.

• Define benchmarks for scientific validity regulatory compliance data quality and operational reliability.

• Establish gold-standard validation datasets and automated evaluation pipelines.

Governance and Standards

• Serve as technical lead for the Evaluation & Standards Board a cross-functional governance body for GenAI quality.

• Present evaluation findings and recommendations to the GenAI Portfolio Steering Committee.

• Define what production-ready means for GenAI in a regulated R&D environment and enforce that standard.

• Set evaluation gates for product development stages: PoC Limited Release Scaled Product and Product Ops.

Cross-Functional Collaboration

• Partner with QMS/MLOps on the handoff from pre-adoption evaluation to production quality monitoring.

• Work with therapeutic area teams on domain-specific evaluation standards.

• Collaborate with Data Strategy & Products on data quality as an input to the evaluation rubric.

• Engage JJIT architecture teams on platform-layer evaluation and security assessments.

Opinion Leadership and Capability Building

• Contribute to the weekly GenAI Outlook newsletter specifically the evaluation implications of frontier AI developments.

• Build internal evaluation capability through training documentation and tooling for teams conducting assessments.

• Publish evaluation standards and rubrics as reference material for the broader organization.

Team Leadership

• Build and lead the GenAI Evaluation & Standards team

• Attract develop and retain top talent in AI evaluation quality science and governance.

• Establish a culture of scientific rigor and independent judgment within the team.

What This Role Is Not

• Not MLOps or production monitoring: the QMS function handles post-deployment operations.

• Not platform engineering.

• Not regulatory affairs or medical writing.

• Not vendor management or procurement: evaluation is independent of purchasing.

Key Qualifications

• Advanced degree (PhD strongly preferred) in computational biology bioinformatics data science computer science AI/ML biomedical engineering applied mathematics or related discipline.

• Minimum 8 years of post-academic industry experience with significant time in pharmaceutical or biotechnology R&D.

• Hands-on expertise with generative AI systems: large language models retrieval-augmented generation agentic frameworks and prompt engineering.

• Demonstrated track record designing evaluation frameworks scientific benchmarks or quality standards for AI/ML systems.

• Strong people leadership experience including building and managing technical teams in a matrixed organization.

• Excellent communication skills: ability to present complex technical findings clearly to non-technical senior stakeholders and defend unpopular conclusions.

• Strategic thinking capability: ability to connect technical evaluation to business impact and organizational decision-making.

Preferred Qualifications

• Experience with AI/ML evaluation in regulated environments (FDA EMA or equivalent).

• Domain expertise in at least one therapeutic area: oncology immunology or neuroscience.

• Experience influencing build-versus-buy decisions at an organizational level.

• Prior collaboration with enterprise IT delivery organizations.

• Experience with evaluation of multi-modal AI systems (text image structured data).

• Familiarity with enterprise data standards and frameworks (CDISC FHIR ontologies).

• Track record of publishing evaluation methodologies or contributing to AI governance frameworks.

Required Skills

Advanced Analytics AI/ML Evaluation Benchmarking Critical Thinking Data Science Evaluation Framework Design Generative AI Inclusive Leadership Leadership Quality Standards RAG Systems Scientific Rigor Strategic Thinking

Preferred Skills

Agentic AI Bioinformatics CDISC Clinical Data Computational Biology Data Governance FHIR Knowledge Graphs Multi-modal AI Ontology Pharmaceutical R&D Regulatory Compliance Therapeutic Area Expertise

#LI-GR

#LI-Hybrid

#JRDDS
#JNJDataScience
#JRD

Required Skills:

Preferred Skills: