Rubric Engineering and Measurement Specialist

About Child Mind Institute

Were dedicated to transforming the lives of children and families struggling with mental health and learning disorders by giving them the help they need. Weve become the leading independent nonprofit in childrens mental health by providing gold-standard evidence-based care delivering educational resources to millions of families each year training educators in underserved communities and developing tomorrows breakthrough treatments.

Position Details:

As part of the Center for Data Analytics Innovation and Rigor team you will report to Technical Project Manager you will lead the implementation and validation of evaluation frameworks for assessing both AI systems in clinical and mental health research contexts and the symptoms of their users. You will own the distillation of clinical definitions into evaluation rubrics combining applied clinical science with evaluation operations to ensure that rubrics are clinically grounded and operationally scalable. You will work closely with both the clinical and data science teams ensuring knowledge is effectively operationalized to the satisfaction of both specializations. Supervising a small team of data science and software professionals you will establish reliability standards oversee the design of metrics for use in multi-rater evaluation systems and ensure scientific rigor through quality assurance versioning and comprehensive documentation.

This is a full-time exempt position. A hybrid schedule will be required for employees based in the New York Tri-State area or California Bay Area with a minimum of four (4) days per week in the office on a schedule determined by your supervisor. Remote positions will be considered for candidates located outside of these regions . The in-office requirement and schedule are subject to change based on the needs of the program and the organization.

Please note: We are not currently able to employ individuals residing in Ohio North Dakota Washington or Wyoming.

You Will:

• Translate clinical symptom evaluations and manualized therapeutic protocols into reliable consistent precise and scorable evaluation criteria.
• Design and implement metadata standards and protocols for multi-rater clinical evaluation systems enabling crowd-sourced ratings from diverse clinical experts.
• Set reliability standards and adjudication logic for resolving disagreements among clinical annotators ensuring consistency across large-scale annotation efforts.
• Establish and enforce quality assurance processes versioning systems and comprehensive documentation for all evaluation rubrics and measurement frameworks.
• Contribute to scientific publications white papers and technical documentation of measurement methodologies and validation studies.
• Collaborate with clinical leadership to ensure evaluation criteria accurately reflects clinical best practices safety standards and clinician feedback.
• Supervise and mentor a small team of data analysts and engineers providing guidance on evaluation methodologies statistical analysis and rubric implementation.
• Collaborate with other technical project stakeholders to ensure that technical definitions and requirements of all curated rubrics and resources are being met.
  
  

You Have:

• PhD PsyD MD or equivalent experience in measurement science psychometrics biostatistics clinical psychology epidemiology or related field
• 8 years of applied evaluation experience in clinical research health outcomes research or psychometric assessment
• 5 years of experience with patient evaluation clinical rating scales or performance assessment frameworks
• 5 years of experience in at least one scientific programming language (e.g. Python/R Matlab) and related toolboxes or frameworks (e.g. Tidyverse Scipy Sklearn Polars Pytorch) is required. Experience can be concurrent with education.
• 5 years of experience with applied statistics including reliability analysis classical test theory item response theory or factor analysis.
• Demonstrated leadership of multi-rater or benchmark systems.
  
  

#LI-hybrid #LI-remote

Our Benefits

Our great compensation package and benefits include medical insurance 401(k) paid parental leave dependent care flexible work schedules discounted tickets and entertainment perks programs. For more information about our benefits please visit our employee benefits website.

Pay Range

The salary range for the position is posted. Factors such as candidates work experience education/training job-related skills internal peer equity as well as market and business considerations affect the salary offered within this addition this salary may be subject to a geographic adjustment (according to a specific city and state and depending on the role) if an authorization is granted to work outside of the location listed in this posting.

EEO Disclaimer

Child Mind Institute is committed to fostering an inclusive and equitable workplace where all individuals are treated with respect and dignity. We are proud to be an equal opportunity employer and prohibit discrimination and harassment of any kind.

We provide equal employment opportunities to all employees and applicants for employment regardless of race color religion creed sex (including pregnancy childbirth breastfeeding or related medical conditions) sexual orientation gender identity gender expression age national origin ancestry citizenship status marital status military or veteran status physical or mental disability genetic information medical condition or any other characteristic protected by applicable federal state or local laws.

In compliance with California law we also prohibit discrimination based on reproductive health decision-making status as a victim of domestic violence sexual assault or stalking or any other category protected by the California Fair Employment and Housing Act (FEHA). In New York we extend this prohibition to include status as a victim of domestic violence familial status or any other characteristic protected by the New York State Human Rights Law (NYSHRL).

Child Mind Institute is dedicated to ensuring accessibility and reasonable accommodations for individuals with disabilities or medical conditions. If you require an accommodation to participate in the application process or perform your job please contact our HR Department at

This policy applies to all aspects of employment including recruitment hiring promotion termination layoff recall transfer leaves of absence compensation benefits and training.