Principal Software Reliability Engineer

About This Role

This is aProduct Reliabilityposition not an infrastructure SRE role. Our DevOps team manages the infrastructure platform; this role focuses on application and service-level reliability working directly with product engineers.

This is the first role of its kind in product engineering. Reporting to the VP of Product Engineering for Consumer Identity youll drive reliability efforts across the team: defining the roadmap prioritizing initiatives and partnering with engineering directors and senior ICs to deliver them.

Why Join Us

• Greenfield opportunity:Youll define Product Reliability as a discipline here. Build the playbook not inherit one.
• High-impact domain:Consumer Identity powers identity verification and biometric authentication for some of the worlds largest financial institutions. Our reliability directly impacts fraud prevention and customer onboarding at scale.
• Real authority:Direct line to VP Engineering budget for tooling seat at architecture council and service reviews.
• Strong foundation:Were not firefighting. 99.98% uptime means youre optimizing not triaging chaos.
• Technical depth:Work across ML pipelines computer vision systems and mobile SDKs (not just YAML and dashboards).
• Ownership culture:Engineers own their services end-to-end; youll amplify that not replace it

Experience Level

Staff SRE

• 8 years in software engineering
• 4 years in reliability/SRE
• Drives reliability initiatives across multiple teams; hands-on with complex systems

Principal SRE

• 15 years in software engineering
• 6 years in reliability/SRE
• Sets technical direction org-wide; influences business-unit-level reliability strategy

Were open to either level. Scope and compensation will match your experience. Principal candidates should demonstrate cross-org impact and a track record of building reliability programs from scratch.

Current State Incident Analysis ()

• Postmortem volume peaked in 2023 down48%since then despite increased release cadence
• P0-to-P1 ratio remains stable despite lower overall incident volume
• 65%change-induced incidents (deployments migrations config changes);35%organic (third-party outages expirations attacks)
• Change-induced ratio improved modestly: 69% 62%
• Detection time:35 min 18 min
• Customer-first detection:40% 22%

Availability Targets

• 2024 & 2025 average uptime:99.98%(as available in our public status page)
• Goal: Consistent99.99%(four nines) average uptime SLO breach reductions

System Simplification

• Were reducing system complexity to narrow the reliability target area:
• Microservices (K8s deployments/rollouts) reduced29%from peak with further cuts planned for 2026
• Goal: Smaller footprint higher reliability lower cost for new regions

Role Objectives

• Primary goal:Improve release safety reduce releases that cause downtime or SLO degradation.
• We already have foundational systems in place:
• Automated test coverage and crowd testing
• A/B testing and dark canaries
• Progressive rollouts (infrastructure and application level)
• Back-testing against historical data
• To consistently exceed four nines we need to mature these systems and build new capabilities.

Ideal Candidate Profile

Mindset

• Passionate about reliability as a discipline not just a checkbox
• Focused on reliability not product features but willing to learn the product to understand impact
• Hands-on: eager to build tooling and systems
• Pragmatic about balancing reliability with development velocity

Required Skills

Software Engineering

• Strong software engineering in at least one of our backend languages (Python Ruby ); able to navigate most of our codebase
• Experience building reliability tooling: progressive delivery automated rollbacks monitoring/alerting

Reliability Patterns

• Deep knowledge of resilience patterns: circuit breakers bulkheads back-pressure retries with backoff rate limiting load shedding graceful degradation
• Solid incident management and blameless postmortem practices

Observability

• Proficiency with observability: distributed tracing structured logging metrics instrumentation
• Uses data to drive decisions: experienced with SLIs SLOs and error budgets

Communication

• Skilled at influencing without authority
• Able to hold deep technical reliability discussions with senior ICs

Nice-to-Have

• Experience with chaos engineering (fault injection game days controlled failure experiments)
• ML system reliability experience (mixed I/O and CPU-bound workloads non-deterministic behavior model serving)
• Familiarity with our specific stack (Datadog Kubernetes AWS GitLab CI/CD)
• Experience leveraging LLMs for code analysis design doc review or automated runbook generation
• On-call experience in a high-availability environment

Our Stack

• Backend:Python Ruby on Rails
• Frontend:React TypeScript
• Mobile:Swift (iOS) Kotlin (Android) React Native
• Infrastructure:AWS Kubernetes Terraform SNS SQS
• Databases:PostgreSQL (Aurora) Redis OpenSearch
• Observability:Datadog Splunk Sentry
• ML:PyTorch TensorFlow
• CI/CD:GitLab (on-prem)

For US roles or where applicable:

Entrust is an EEO/AA/Disabled/Veterans Employer

For Canadian roles or where applicable:

Entrust values diversity and inclusion and we are committed to building a diverse workforce with wide perspectives and innovative ideas. We welcome applications from qualified individuals of all backgrounds and we strive to provide an accessible experience for candidates of all abilities.

If you require an accommodation contact .

Recruiter: