Engineering & Data Science Specialist, Applied AI Evaluations

Address:

Job Family Group:

About the Team

BMOs Applied AI team sets the standard for safe high-performing AI systems across the bank building BMOs leading-edge AI products and capabilities. Our work enables BMOs strategic outcomes bridging business needs with scalable and responsible AI. If you thrive on technical challenges and want to make a strategic impact this role is for you.

About the Role

As an Engineering & Data Science Specialist in Applied AI Evaluations you will design implement and optimize evaluation frameworks and pipelines that measure the quality safety and performance of advanced AI systems at BMO. Youll work closely with the Product Owner and collaborate with engineering research and business teams to translate evaluation standards into robust scalable solutions.

Your work will directly impact model development deployment and continuous improvement cyclesensuring that evaluation signals are deeply integrated into experimental and production systems. Youll contribute to the development of reusable tools agents and harnesses that support responsible AI practices and regulatory compliance.

Key Responsibilities

• Develop implement and maintain evaluation pipelines and harnesses for AI models ensuring reliability reproducibility and scalability for banking applications.
• Collaborate with the Product Owner to translate evaluation standards and business requirements into technical solutions.
• Prototype and validate evaluation methods using real banking workflows integrating feedback into model training and deployment.
• Integrate evaluation metrics and signals into research training and production systems to drive measurable improvements in model performance and customer outcomes.
• Partner with engineering research and product teams to shape model interaction paradigms and deployment strategies.
• Build reusable tools and systems that enable contributions from across BMO and raise the quality bar for AI solutions.
• Ensure all evaluation frameworks adhere to regulatory compliance and Responsible AI requirements.
• Document processes share knowledge and contribute to a culture of continuous learning and responsible innovation.

Qualifications

• 3 years of experience in engineering or data science roles preferably in AI machine learning or data-driven product environments.
• Strong programming skills (e.g. Python Java or similar) and experience with ML/AI frameworks (e.g. PyTorch TensorFlow).
• Experience building and maintaining evaluation pipelines for large language models (LLMs) and/or other AI systems including multi-agent workflows tool use or long-context scenarios.
• Solid understanding of deep learning concepts model training and deployment practices.
• Familiarity with MLOps CI/CD and cloud-based AI infrastructure is a plus.
• Strong analytical and problem-solving skills with the ability to communicate technical concepts to both technical and non-technical audiences.
• Motivated by high-impact collaboration and able to thrive in fast-paced ambiguous environments.
• Commitment to BMOs values of inclusion integrity and responsible innovation.

Salary:

Pay Type:

The above represents BMO Financial Groups pay range and type.

Salaries will vary based on factors such as location skills experience education and qualifications for the role and may include a commission structure. Salaries for part-time roles will be pro-rated based on number of hours regularly worked. For commission roles the salary listed above represents BMO Financial Groups expected target for the first year in this position.

BMO Financial Groups total compensation package will vary based on the pay type of the position and may include performance-based incentives discretionary bonuses as well as other perks and rewards. BMO also offers health insurance tuition reimbursement accident and life insurance and retirement savings plans. To view more details of our benefits please visit: Us

At BMO we are driven by a shared Purpose: Boldly Grow the Good in business and life. It calls on us to create lasting positive change for our customers our communities and our people. By working together innovating and pushing boundaries we transform lives and businesses and power economic growth around the world.

As a member of the BMO team you are valued respected and heard and you have more ways to grow and make an impact. We strive to help you make an impact from day one for yourself and our customers. Well support you with the tools and resources you need to reach new milestones as you help our customers reach theirs. From in-depth training and coaching to manager support and network-building opportunities well help you gain valuable experience and broaden your skillset.

To find out more visit us at is proud to be an equal employment opportunity employer. We evaluate applicants without regard to race religion color national origin sex (including pregnancy childbirth or related medical conditions) sexual orientation gender identity gender expression transgender status sexual stereotypes age status as a protected veteran status as an individual with a disability or any other legally protected characteristics. We also consider applicants with criminal histories consistent with applicable federal state and local law.

BMO is committed to working with and providing reasonable accommodations to individuals with disabilities. If you need a reasonable accommodation because of a disability for any part of the employment process please send an e-mail to and let us know the nature of your request and your contact information.

Note to Recruiters: BMO does not accept unsolicited resumes from any source other than directly from a candidate. Any unsolicited resumes sent to BMO directly or indirectly will be considered BMO property. BMO will not pay a fee for any placement resulting from the receipt of an unsolicited resume. A recruiting agency must first have a valid written and fully executed agency agreement contract for service to submit resumes.