AI Quality and Evaluation Lead

We are hiring an AI Quality and Evaluation Lead Application Development!Reporting To:Assistant Vice President IT Application DevelopmentFull-Time/Part- Time:Full-timePosting Date:August 21 2026Closing Date:September 21 2026Hours of Work:8:30 a.m. 5:00 :Office Location: 14.4TorontoGreat location! Steps away from the main public transit station What we offer:Highly competitive compensation package which includes base salary bonus benefits and career advancement opportunities! *Eligibility for benefits is dependent on the terms of employmentThe Opportunity:The AI Quality and Evaluation Lead is accountable for the evaluation strategy quality standards release gates guardrails and continuous monitoring used across First National AI Factory role defines what good enough to ship means forAI enabledmortgage workflows and converts business policy risk appetite user expectations and production evidence into measurable acceptancecriteria. Working within the AI Platform Squad leadshands on evaluation for residential underwriting use cases with the mandate to evolve a scalable quality capability as the AI Factory grows including reusable test assets automated continuous evaluation consistentscorecards and clear accountability between product squads and shared quality teams. How you will contribute:Own and continuously evolve the AI Factory quality and evaluation strategy including risk tiers standard metrics acceptance thresholds representative test suites release evidence monitoring expectations and continuous improvement practices across product use case specific quality measures with the Product Manager the Residential Underwriting Business SME the AI Engineering Lead QA Engineers Risk Compliance and Security covering policy adherence factual grounding completeness precision recall false positives false negatives confidence explainability user impact and appropriate the methodology for evaluation harnesses golden datasets adversarial tests regression suites human review protocols and automated scoring workflows across document intelligence retrieval summarization agent tool use deterministic calculations andmulti stepworkflows with the AI Platform Squad building and operating the underlying evaluation the quality and evaluation gates within First National production readiness standards jointly with the AI Engineering Lead who owns engineering and operational readiness including UAT and parallel run evidence minimum performance thresholds guardrail validation privacy and access control checks rollback criteria kill switch readiness runbooks and approval collaboration with Information Security conduct failure mode testing for prompt injection data leakage unsupported claims policy deviations tool misuse unsafe write back inappropriate automation model degradation and other AI specific operational production quality using evaluation results model and prompt versions drift latency user feedback exception rates override patterns incidents and token or platform cost and define the triggers for investigation rollback retraining prompt changes or workflow defect taxonomies and root cause practices that distinguish model retrieval prompt data rules integration user experience and process issues and work with product and engineering teams to prioritize durable the function from hands on evaluation of the first use cases to a federated operating model with reusable scorecards automated continuous evaluation trained squad level QA practicesoversight of AI Quality Analysts as volume grows independent model and vendor benchmarking and audit ready quality experience you need:Bachelors degree in computer science engineering data science statistics mathematics informationsystemsora relateddiscipline.7 plus years of progressive experience in quality engineering software testing model validation machine learning data science risk analytics AI governance or related technology designing evaluation frameworks and release criteria forproductionAI or generative AI systems including RAG document intelligence agents model and prompt lifecycle management observability and human in the loop analytical and technical skills with Python SQL test automation data analysis statistical methods and evaluation tooling such asMLflow RagasDeepEval or comparable knowledge of accuracy and retrieval metrics confidence scoring sampling experiment design regression testing drift monitoring adversarial testing red teaming guardrails privacy security and operational risk to translate business policy and risk appetite into measurable acceptance criteria and to explain quality risk and trade off implications clearly to technical business control audit and senior building or scaling a quality capability through reusable standards automated test assets continuous monitoring governance evidence coaching and leadership of analysts or cross functional quality communities including exposure to model risk credit risk or independent validation in financial services mortgage lending lending operations servicing broker channels third party partnerships or regulated technology environments is :External:Engages implementation partners cloud providers data and AI platform providers model providers and specialist vendors on evaluation methods independent benchmarking quality evidence and knowledge transfer to First :Works within the AI Platform Squad whilemaintainingassessment independence from the delivery teams whose work it Environment and Physical Demands Analysis:Office environmentPeriods of high volume with tight timelinesLong periods of stationary position/sittingProlonged periods of repetitive movement (i.e. using a keyboard and mouse)Long periods of time in viewing a computer screenMulti-tasking may include speaking to customers on a telephone call while looking up information on a computer join First NationalCompetitive CompensationComprehensive benefits program (i.e. Health Spending Account Maternity and Parental Leave Top Up)Extensive training programs to set our employees up for success Modern office environment conducive to collaborationSupportive teamwork culture Opportunities to give back to the communities and work through events focused on a variety of charities Ongoing social events throughout the yearThe team youll join: Founded in 1988 First National is one of Canadas largest non-bank lenders. We provide residential mortgages exclusively through the mortgage broker channel and we are Canadas largest commercial mortgage lender. First National has been consistently recognized as a great place to work and we are proud that our employee engagement feedback is higher than our industry partners. We would like to thank all applications for their interest in this existing vacancy but only candidates selected for an interview will be contacted. Artificial Intelligence is not used in our recruitment or hiring process for this National is proud to be an equal opportunity employer and is committed to diversity and inclusion regardless of race color religion national origin age gender identity physical or mental disability sexual orientation and any other category protected by National supports requests for accommodation from applicants with disabilities; please contact Human Resources at should you need an accommodation at any point in the recruitment process.#FNLOON