AI Evaluation Specialist (CS-2)

Anything is possible at the NRC named in 2025 one of Canadas Top Employers for Young People Top Employer in the National Capital Region and Forbes Canadas Best Employers!

As Canadas largest research and innovation organization our world-renowned research pushes the boundaries of science and engineering to make the impossible possible. Every day we explore new ideas through innovative research and help companies discover possibilities that impact Canadas future and the world.

At the NRC youll also discover new possibilities. Our supportive workplace fosters a culture of creativity welcoming fresh perspectives and innovation at all levels. We value teamwork. Youll collaborate across multiple fields and with the brightest minds to find creative solutions. Most importantly youll discover whats possible within you as you grow make valuable contributions and progress in your professional journey. From ground-breaking discoveries to a life-changing career discover your possible at the NRC.

Are you passionate about safe and responsible AI Do you want to be part of a pioneering team shaping how AI systems are evaluated measured and deployed safely Do you want your work to have a real impact for Canada and Canadians If so we want to meet you!

We are looking for a CS-02 AI Evaluation Specialist. Come join our dynamic team at the National Research Council (NRC) as we embark on ground-breaking projects related to AI safety and security. Be part of a pioneering team that will establish a new AI Safety Lab for conducting system evaluations and translating technical findings into practical solutions and recommendations for AI practitioners and policy makers.

Under the guidance of the AI Safety Lab Lead you will play a critical role in building maintaining and using a new infrastructure for AI safety evaluation. You will also set up run and monitor advanced agentic evaluation at scale on frontier AI models while contributing to cutting-edge AI safety research conducted by the NRC in collaboration with national and international partners.

Key Responsibilities include:

• Build maintain and use a state-of-the-art model evaluation infrastructure combining on-premises and cloud-based clusters.
• Support AI safety researchers leveraging the evaluation infrastructure.
• Set up run and monitor large-scale evaluations involving multiple AI models agents and benchmarks.
• Steward and document protocols for model evaluation and experiment tracking.
• Contribute to develop and maintain customized evaluation benchmarks.
• Contribute to the analysis and reporting of model evaluation results.
• Stay current with the latest advancements in model evaluation tools approaches and packages.

Applicants must demonstrate within the content of their application that they meet the following screening criteria in order to be given further consideration as candidates:

University degree in computer science computer engineering or related field.

Education Assets

• A specialization in data analytics machine learning or Artificial Intelligence (AI) may be considered an asset.
• A Masters degree in a field related to the position may be considered an asset.

Equivalency
A college diploma with significant experience directly related to the duties of this position may be considered as an educational equivalent.

For information oncertificates and diplomas issued abroad please see Degree equivalency

You must meet the essential experience criteria and at least two of the Specialised Technical Experience criteria to be considered in this process.

Essential

• Experience developing Python-based tooling and automation for ML/AI workflows in a research or production environment.
• Experience with the programmatic use and orchestration of large language models (both closed-source and open-source).
• Experience with containerization technologies (such as Docker Podman or Apptainer).
• Experience with high-performance computing (HPC) clusters and job orchestration (e.g. SLURM).
• Experience documenting experimental AI results through technical reports internal documentation or scientific publications.

Specialised Technical Experience

• Experience setting up running or contributing to evaluations or benchmarking of AI systems.
• Experience with red-teaming workflows adversarial testing or stress-testing of AI systems (including agentic systems).
• Experience with AI/ML experiment tracking tools (e.g. MLflow Weights & Biases).
• Experience developing or curating AI benchmark datasets and associated documentation (e.g. through HuggingFace).

Reliability Status

For a Reliability Status verification of background information over a period of 5 years is required.

English

Information on language requirements and self-assessment tests

Candidates will be assessed on the basis of the following criteria:

You must meet all of the essential competencies and at least one of the Specialised Technical Competency criteria to move forward in this process.

Essential Technical Competencies

• Ability to design and implement reproducible AI workflows using Python.
• Ability to make sound technical decisions when using different models accounting for factors such as API constraints compute budgets reproducibility and result validity.
• Ability to diagnose and resolve infrastructure issues in HPC or cloud-based evaluation environments including resource contention job failures and performance bottlenecks.
• Knowledge of AI safety evaluation frameworks (e.g. Inspect AI Moonshot) and their application to frontier model assessment.
• Knowledge of complex data storage and querying tools for handling large volumes of evaluation data.
• Knowledge of software engineering best practices including version control CI/CD pipelines automated testing and related modern software development tools (e.g. Git GitLab/GitHub) and practices.

Specialised Technical Competencies

• Knowledge of ML experiment tracking tools (e.g. MLflow Weights & Biases) and their integration into reproducible research workflows.
• Knowledge of agentic AI frameworks and protocols.
• Familiarity with AI safety concepts risk taxonomies or governance frameworks (e.g. TBS guidelines NIST AI Risk Management Framework model evaluation practices).

For this position the NRC will evaluate candidates using the following competency profile(s): Management Services; Research Technician/Technologist

View all competency profiles

From $86503to $108068per annum.

NRC employees enjoy a wide-range of competitive benefits including a robust pension plan comprehensive health and dental coverage disability and life insurance office closure at the end of December and additional supports to enhance your well-being throughout your career and beyond.

• In 2025 the NRC was chosen as one of Canadas Top Employers for Young People a National Capital Region Top Employerand Forbes Canadas Best Employer.
• Relocation assistance will be determined in accordance with the NRCs directives.
• A pre-qualified listmay be established for similar positions for a one year period.
• Preference will be given to Canadian Citizens and Permanent Residents of Canada. Please include citizenship information in your application.
• The incumbent must adhere to safe workplace practices at all times.
• We thank all those who apply however only those selected for further consideration will be contacted.