Principal Engineer, AI Platform

Address:

Principal Engineer AI Platform & Fabrics

Description

BMO is building the platform capabilities that make enterprise AI safe governed and scalable. We are seeking experienced Principal/Senior engineers tobuild and operate the core infrastructure that governs how AI runs at BMO: the AI Gateway Policy Engine Identity Fabric AI Registry Guardrails Runtime and AI Observability.

This is abuild-and-runengineering role. You will own capabilities end to end; designing shippingand operatingthem in production including on-call. You will not build the AI models or applications themselves (those are domain-owned); you build thegoverned platform they run onand the runtime evidence that proves they run within policy across AWS Azure and Microsoft AI surfaces under OSFI and OCC expectations.

You are a hands-on engineer who has built shared platform services at scale cares deeply about operability latency and correctness and understands that in a regulated bank the infrastructure mustproduce its own evidence. You are energized by taking real engineering assets that includes an existing developer portal an AI registry a body of policy-as-code and gateway integrations and hardening scaling and governing them into enterprise-grade platform capabilities. You raise the technical bar for those around you and mentor as you build.

What Youll Build & Operate

Depending on your specialization you will own one or more of the following capability areas:

Enterprise Control Plane

• Portal & Registry a federated AI Registry (agents models tools channels evaluations across 16 asset types) with self-service onboarding and lifecycle workflows; federation with external registries (Agent 365 AgentCore MLflow).

• Policy Engine policy-as-code infrastructure (Cedar/OPA) a policy compilation pipeline GitOps-based domain-scoped bundle distribution risk-tiered approval workflows and a policy simulation sandbox.

• Observability & Audit a multi-pipeline telemetry architecture (operational security compliance) OpenTelemetry GenAI conventions cross-pipeline trace correlation lineage-stamped traces and a 7-year tamper-evident audit lake producing regulator-ready evidence.

• Governance & Lifecycle certification workflows automated compliance scoring decommission governance and evidence generation for architecture and model-risk review.

Domain Orchestration

• Gateway Runtime domain-hub deployment across AWS and Azure; an inline enforcement engine performing request-time policy evaluation routing residency budget/quota and circuit breaking within strict tiered latency budgets (Fast <5ms / Standard <25ms / Full <50ms); cross-region failover.

• Guardrails Runtime a multi-stage safety pipeline (input moderation prompt-injection defense PII output validation hallucination detection policy enforcement) with bilingual EN/FR parity and behavioral guardrails for agentic workloads (goal hijacking intent drift excessive agency).

• Identity Fabric workload identity for AI (SPIFFE/SPIRE) token-exchange bridging per-domain trust boundaries Entra Agent ID integration on-behalf-of identity propagation and cross-cloud token federation with zero-trust attestation.

What Youll Do

• Own capabilities end to end design implement test ship and operate production infrastructure including on-call ownership of what you build (no separate run team).

• Engineer foroperability and defensibility from day one instrumentation SLOs latency budgets failure modes and runtime evidence built in not bolted on.

• Build theAPIs SDable interfaces and integrationsthrough which domains DevOps pipelines and enterprise systems consume platform capabilities.

• Implementpolicy enforcement guardrails identity attestation and auditas first-class engineering concerns correct performant and provable.

• Ensure every capabilityproduces runtime evidenceconnecting AI activity to policy enforcement identity and lineage for model-risk and regulatory review (OSFI E-23 OCC).

• Assess emerging AI infrastructure foundation-model access patterns and standards; make deliberate cost-aware engineering choices.

• Mentor and raise the bar set engineering standards review designs and code and grow depth across the team.

• Partner closely withAI Developer Experience(so domains can consume what you build)AI SecurityandAI SDLC(embedded specializations) and theSenior AI Architect(architectural coherence).

Education & Experience

• Bachelors degree in Computer Science Software Engineering or a related technical discipline (Masters preferred).

• 8 years of software/platform engineering experience (Principal) or 5 years (Senior) with substantial timebuilding and operating shared platform servicesat enterprise scale.

• Demonstrated experience operatingproduction infrastructure with real SLOs and on-callownership ideally in a regulated industry (financial services strongly preferred).

• Depth in one or more of:API gateways / traffic enforcement; policy-as-code and authorization; workload identity / zero-trust; observability and telemetry pipelines; audit/compliance data platforms.

Required Core Skills

Platform Engineering Depth

• Strong distributed-systems and platform-engineering fundamentals: latency-sensitive request paths resilience patterns (circuit breakers failover) multi-tenancy and high availability.

• Strong programming skills (Python and/or Go preferred; TypeScript/Java an asset) for building performant services APIs and integrations.

• Cloud-native architecture acrossAWS and Azure: containers/Kubernetes service mesh and Infrastructure as Code (CDK Terraform CloudFormation/ARM).

• Robust CI/CD GitOps and DevSecOps practice; Git-based workflows (Bitbucket/GitHub) Jira Confluence.

Capability-Specific Depth (one or more)

• Policy/Authorization:Cedar OPA/Rego policy compilation and distribution risk-tiered approval workflows.

• Identity/Zero-Trust:SPIFFE/SPIRE mTLS token exchange OAuth/OIDC federated and cross-cloud identity Entra ID/Agent ID OBO.

• Observability/Audit:OpenTelemetry (incl. GenAI conventions) distributed tracing Dynatrace/Splunk or equivalents tamper-evident/immutable audit stores data lineage.

• Gateway/Guardrails:API gateway internals inline enforcement LLM routing/abstraction prompt-injection and PII defenses hallucination detection AI evaluation.

• Registry/Portal:service catalogues asset registries lifecycle workflows federation with external registries.

GenAI & Governance Context

• Working knowledge of GenAI platform patterns: LLM/AI gateways RAG and agentic patterns foundation models embeddings and guardrails sufficient to build the infrastructure they depend on.

• Familiarity with AI/ML platforms (Bedrock Azure OpenAI SageMaker MLflow) and orchestration frameworks (LangChain LlamaIndex).

• Grounding in Responsible AI AI/data governance privacy cloud security and IAM as applied to AI workloads.

Certifications (Preferred)

• AWS Certified Solutions Architect (Associate/Professional) / ML Specialty

• Microsoft Certified: Azure Solutions Architect Expert / Azure AI Engineer Associate

• Kubernetes (CKA/CKAD); HashiCorp Terraform Associate

• Security/identity certifications (relevant to Identity Fabric roles)

Other Skills

• Strong communication and collaboration across engineering security architecture and domain teams.

• A critical thinker with strong analytical and problem-solving skills.

• Self-directed comfortable with ambiguity and a fast-evolving mandate.

• Able to deliver complex work under tight timelines; participates in on-call rotation for owned services.

The above represents BMO Financial Groups pay range and type.

Salaries will vary based on factors such as location skills experience education and qualifications for the role and may include a commission structure. Salaries for part-time roles will be pro-rated based on number of hours regularly worked. For commission roles the salary listed above represents BMO Financial Groups expected target for the first year in this position.

BMO Financial Groups total compensation package will vary based on the pay type of the position and may include performance-based incentives discretionary bonuses as well as other perks and rewards. BMO also offers health insurance tuition reimbursement accident and life insurance and retirement savings plans. To view more details of our benefits please visit: Us

At BMO we are driven by a shared Purpose: Boldly Grow the Good in business and life. It calls on us to create lasting positive change for our customers our communities and our people. By working together innovating and pushing boundaries we transform lives and businesses and power economic growth around the world.

As a member of the BMO team you are valued respected and heard and you have more ways to grow and make an impact. We strive to help you make an impact from day one for yourself and our customers. Well support you with the tools and resources you need to reach new milestones as you help our customers reach theirs. From in-depth training and coaching to manager support and network-building opportunities well help you gain valuable experience and broaden your skillset.

To find out more visit us at is committed to an inclusive equitable and accessible workplace. By learning from each others differences we gain strength through our people and our perspectives. Accommodations are available on request for candidates taking part in all aspects of the selection process. To request accommodation please contact your recruiter.

Note to Recruiters: BMO does not accept unsolicited resumes from any source other than directly from a candidate. Any unsolicited resumes sent to BMO directly or indirectly will be considered BMO property. BMO will not pay a fee for any placement resulting from the receipt of an unsolicited resume. A recruiting agency must first have a valid written and fully executed agency agreement contract for service to submit resumes.