Senior Scientist GenAI Evaluation

At Johnson & Johnsonwe believe health is everything. Our strength in healthcare innovation empowers us to build aworld where complex diseases are prevented treated and curedwhere treatments are smarter and less invasive andsolutions are our expertise in Innovative Medicine and MedTech we are uniquely positioned to innovate across the full spectrum of healthcare solutions today to deliver the breakthroughs of tomorrow and profoundly impact health for more at .

As guided by Our Credo Johnson & Johnson is responsible to our employees who work with us throughout the world. We provide an inclusive work environment where each person is considered as an individual. At Johnson & Johnson we respect the diversity and dignity of our employees and recognize their merit.

Job Function:

Job Sub Function:

Job Category:

All Job Posting Locations:

Job Description:

At J&J we are building Generative AI solutions to support pharmaceutical R&D literature review evidence synthesis document Q&A therapeutic area knowledge search translational science workflows and R&D decision support. These systems need to be evaluated before teams rely on them in scientific pharma a useful AI response depends on the question user source material therapeutic area and risk of error so quality must be measurable repeatable traceable and scientifically defensible.

As a Senior Scientist in our Generative AI Evaluation & Standards (EQS) function within Data Science and Digital Health you will design build and run the evaluation methods we depend on to assess these systems across R&D. You will author rubrics curate benchmark and golden datasets validate AI judges and produce the quality readouts that inform release decisions. Want to shape how a top pharmaceutical company determines whether its AI is ready for real scientific work This is that role!

KEY RESPONSIBILITIES:

We need someone who can build the evaluation assets our teams count on and continuously improve them based on what we learn. Day to day you will:

• Design build and maintain automated evaluation pipelines for LLM quality RAG performance agent reliability safety and scientific accuracy.
• Author evaluation rubrics and scoring criteria curate golden and synthetic datasets with domain experts and maintain our registry of reusable evaluation assets.
• Validate AI judges against human expert agreement and run model prompt retriever and agent benchmarks that produce standardized quality readouts.
• Analyze failure patterns hallucination unsupported claims weak traceability and turn findings into actionable recommendations.
• Develop therapeutic-area-specific evaluation criteria with scientific clinical and regulatory partners refining them based on real-world feedback.
• Design evaluation methods for scientific reasoning evidence synthesis and hypothesis quality where generic benchmarks fall short.
• Build evaluation tooling and reusable patterns that enable other teams to self-serve.

QUALIFICATIONS

Education:

Masters degree in AI/ML Computer Science Data Science Computational Biology Bioinformatics Biomedical Engineering Applied Mathematics Biostatistics or a related field required. PhD preferred.

EXPERIENCE AND SKILLS:

Required:

We are looking for someone with 6 years of hands-on experience in AI/ML evaluation or data science (Masters) or 3 years of industry experience (PhD). You should have experience designing and running evaluation frameworks scientific benchmarks or quality assessments for AI/ML systems and hands-on work with generative AI large language models retrieval-augmented generation agentic frameworks and prompt engineering. We also value strong proficiency in Python and modern AI/ML tooling (evaluation harnesses embedding models vector databases LLM APIs) the ability to translate expert scientific judgment into measurable criteria rubrics and reproducible protocols and a collaborative self-driven approach to working across multidisciplinary teams.

PREFERRED:

We would love to find someone who also brings experience with AI/ML evaluation in regulated environments (FDA EMA or equivalent) understanding of the drug development pipeline and biomedical data types or domain expertise in oncology immunology or neuroscience. Experience designing or validating LLM-as-judge systems implementing CI/CD evaluation pipelines or publications in AI evaluation NLP or biomedical informatics are all a plus.

OTHER:

English proficiency is required (written and verbal). This is a hybrid role based in Madrid or Barcelona with limited travel (<10%) primarily within Europe.

Required Skills:

Preferred Skills:

The anticipated base pay range for this position is:

Benefits:

In addition to base pay we offer the following benefits*: an annual bonus with set target (% of pay) depending on pay grade / location where the actual amount is based on the employees and companies performance of the previous calendar year or sales commissions. Moreover we offer vacation days parental leave for a minimum of 12 weeks bereavement leave caregiver leave volunteer leave well-being reimbursement programs for financial physical and mental health. We also offer service anniversary and recognition awards and subject to the terms of their respective plans employees - and in some locations eligible dependents - can participate in several insurance plans. For more information visit Employee benefits Supporting well-being & career growth Johnson & Johnson Careers.

*This is for informative purposes only. Amounts and actual benefits may vary by location and are subject to change.