Clinical AI Engineer

We are seeking a Clinical AI Engineer to join our innovative team working at the intersection of healthcare and AI. This role focuses on designing evaluating and improving LLM-powered tools that support abstraction and data analysis for clinical registries. The ideal candidate combines clinical domain expertise with a deep understanding of AI prompt engineering model evaluation and data quality analysis to build scalable solutions for real-world healthcare workflows.

As a Clinical AI Engineer youll work closely with product engineering and clinical operations teams to define test and refine AI behaviors across a portfolio of registry automation tools. Youll design prompts evaluate AI outputs and build systems that ensure safe accurate and performant abstraction across diverse patient populations and registry types.

Responsibilities:

Design and Evaluation of LLM Systems

• Develop and refine prompt strategies tailored to clinical registry questions and workflows.
• Design evaluation protocols and behavioral test suites to assess the performance of LLM-based systems for accuracy relevance and clinical appropriateness.
• Partner with ML/engineering teams to test novel prompting techniques chain-of-thought reasoning and retrieval augmentation in abstraction workflows.

Clinical and Registry Expertise

• Translate complex registry data requirements into structured abstraction workflows using LLMs.
• Apply clinical reasoning and judgement to identify subtle errors gaps and contextual misunderstandings in AI-generated outputs.
• Leverage experience with clinical data abstraction processes and EHR documentation to improve real-world usability and accuracy.

Data Quality Analysis and Measurement

• Design metrics and dashboards to measure model coverage abstraction precision/recall and user correction rates.
• Build error taxonomies and track failure modes across registry products to drive continuous improvement.
• Collaborate with engineering and research teams to recommend architectural or training enhancements to LLM systems.
  
  

Product and Team Collaboration

• Contribute clinical insight and feedback to roadmap development for registry abstraction and data quality tooling.
• Work closely with nurses abstractors and clinical informaticists to validate abstraction behavior and usability.
• Pair with software engineers and AI researchers to test product hypotheses and bring features to production.

What youll need:

Clinical Knowledge & Experience

• Strong foundation in interpreting patient data including diagnoses labs procedures and treatment timelines.
• Experience working directly with registry programs (e.g. STS NSQIP ACC etc.) and abstracting clinical data.
• Familiarity with nursing workflows and how clinicians interact with EHRs and abstraction interfaces.

AI Prompt Engineering and Evaluation

• 1 year hands-on experience designing prompts for LLMs in a professional or research setting.
• Strong intuition for how AI systems interpret instructions and when/how they fail in clinical contexts.

Experience designing test sets and evaluation pipelines for AI output behavior.

Data QA and Measurement

• Experience measuring performance of AI or human-in-the-loop systems with precision recall and coverage metrics.
• Familiarity with clinical data QA principles and constructing reproducible test environments.

Ability to create and use error taxonomies to drive iterative improvement of AI behaviors.

Additional Skills

• Excellent written and verbal communication with both technical and clinical stakeholders.
• Comfortable working in fast-paced cross-functional teams where responsibilities may span research engineering and operations.

Bachelors degree in a clinical field (e.g. Nursing Health Informatics Public Health) or equivalent experience. Masters degree or relevant AI coursework is a plus.

Why we love Carta Healthcare and why you will too!

• Working on a new challenging problem space with LLMs/AI/Healthcare
• Industry leading products
• Work hard and have fun doing it
• Work alongside some of the most talented and dedicated teammates
• Mission driven
• Competitive benefits package including great healthcare benefits and 401k

All applicants require legal authorization to work in the United States now or in the future without sponsorship.

Salary Range: $89900 to $97900