Senior Software Engineer SRE, Retail and Pharmacy

Position Summary:

About the Team

Our Site Reliability Engineering team is the execution engine behind the reliability availability and performance of distributed store technology powering thousands of retail and pharmacy locations nationwide. We operate across pharmacy platforms Point of Sale (POS) systems handheld devices store servers dispensing systems and edge computing infrastructure spanning hybrid cloud and on-premises environments at massive fleet scale.

Our engineering philosophy is grounded in five pillars:DetectionPreventionRecoveryLearning Loops andDeveloper Experience (DevX).

Our operating principle is thereliability covenant: our success is not measured by incident response volume it is measured by the reliability capability we transfer to the engineering teams we serve.Your success in this role is measured by what the engineering teams in your domain can do independently after working with you not by how indispensable you become to them. An SSE who has enabled a development team to detect respond to and learn from production failures without SRE involvement has delivered the highest-value outcome this role can produce.

We trackoperational toil as an engineering metric. Engineers at this level are expected to identify recurring manual work eliminate it through automation document the reduction and treat toil accumulation as a reliability risk not as a sign of operational expertise.

About the Role

As aSenior Software Engineer SRE you independently own the reliability posture of an assigned engineering domain. You are not waiting for direction you are setting it for your domain. You design the alerting strategy own the SLO health lead incident command for production issues facilitate postmortems tune anomaly detection models and partner directly with engineering domain owners to shift reliability left into design.

You are a technical mentor to SE-level engineers and an escalation resource during active incidents. You have the technical depth to diagnose complex distributed system failures the data instincts to distinguish genuine anomalies from noise in ML-generated signals and the organizational skills to drive reliability practice adoption in teams that did not necessarily ask for SRE involvement.

Scope:Domain ownership you operate independently and influence adjacent engineering teams.

The Environment You Are Joining

This role exists inside an active SRE transformation. The domain-based SRE ownership model you will operate within is in its early stages. Some of the toolchains you will work with are being built in parallel with the operational work. Engineering domain owners are simultaneously learning what SRE can offer them.

This is an honest description of the role not a caveat.The opportunity is to shape a domains reliability posture from the ground up in a large-scale consequential technology environment. The engineering decisions you make will affect pharmacy dispensing prescription fill workflows and store operations across thousands of locations. If you are energized by the combination of technical depth and organizational building the scope here is significant. If you are looking for a mature fully-defined SRE environment where the frameworks and toolchains already exist this role will feel like a different challenge.

Success in this role requirespatience alongside technical rigor: you will demonstrate value before you demand process change build credibility before you expect adoption and earn trust with engineering domain owners through partnership rather than mandate.

The operating environment includes anedge computing fleetdeployed directly inside store locations unattended nodes where deployment blast radius is geographic and fleet-wide not functional and service-scoped. You will develop a fleet operations mindset: the primary failure mode in this environment is deployment and configuration propagation not service logic.

What You Will Do

Detection & Observability

• Own SLI/SLO health for your assigned domain end-to-end; monitor error budget burn rates and drive proactive burn-down actions before incidents reach end users or pharmacy patients

• Design multi-signal alerting strategies that go well beyond threshold alerts: burn rate alerting composite health signals andanomaly-based detection using time-series models and validate their output against production ground truth

• Tune and maintain ML-based anomaly detection modelsin production: adjust sensitivity thresholds evaluate false positive rates against alert fatigue metrics and iterate on model configurations based on incident data

• Build Critical User Journey (CUJ)-anchored dashboards that surface end-to-end business flow health not just individual service metrics

• Drive alert signal-to-noise improvement across the domain; own suppression policies during deployments and maintenance windows to protect on-call quality

• Identify and instrument unmonitored external dependencies third-party APIs downstream services data providers using tools likePrometheus Blackbox ExporterOpenTelemetry or custom health probes

• Contribute to the observability platform design: understand the hot/warm/cold tier architecture (Apache Kafka ClickHouse cold storage) and contribute domain-specific data models and SLI schemas

Prevention & Reliability Engineering

• Lead Production Readiness Reviews (PRR) for services in your assigned domain; own the readiness gate sign-off and be accountable for what makes it into production on your watch

• Design and execute fault injection experiments at service level using tools such asLitmusChaosChaos Toolkit orGremlin; validate blast radius assumptions before rollouts reach production

• Partner with engineering domain owners on reliability requirements during architecture design and sprint planning reliability is designed in not bolted on

• Owndependency risk mappingfor your domain: catalog third-party API failure modes shared infrastructure failure paths and chain-wide blast radius scenarios

• Apply afleet operations mindsetto change management: for any configuration or deployment change touching the edge fleet assess deployment blast radius by node cohort review the rollback procedure and contribute to the go/no-go decision on high-risk changes

• Understand progressive rollout strategies canary cohorts staged fleet expansion blast radius budgets and apply them in high-risk deployment reviews

Incident Response & Recovery

• Serve as the primary on-callTechnical Incident Commander (IC)for domain incidents; drive structured bridge calls from detection to resolution using established incident command frameworks

• Author and maintain P0/P1 runbooks with validated step-by-step remediation procedures; own a quarterly review and dry-run testing cadence to ensure runbooks are accurate when they are actually needed

• Lead post-incident reviews using structured root cause analysis: causal chain documentation origin layer classification and contributing factor identification not just a timeline of what happened

• Serve as the real-time escalation point and technical decision support for SE engineers during active incidents

• Pursue Technical Incident Commander (TIC) certification; qualify as a cross-domain IC candidate available to lead major incidents beyond your assigned domain

Learning Loops & Continuous Improvement

Learning Loops at this level issystems engineering for organizational memory not postmortem administration.

• Facilitate domain-level postmortems with rigor and structure: timestamped timelines contributing factor taxonomy (origin layer failure pattern classification) systemic findings and action items with owners due dates and measurable definitions of done

• Measure learning velocity: track not just whether action items close but whether the incident class frequency decreases after a fix is applied. Report the ratio of incident classes that recurred versus incident classes that were resolved systemically. This is your primary learning loop quality metric

• Identify recurring incident patterns across your domain and drive architectural or process changes that eliminate the root cause not suppress the symptom

• Close the prevention loop: translate postmortem findings directly into PRR requirement updates new monitoring coverage runbook improvements or development-team practices that prevent recurrence. A postmortem that doesnt change something upstream hasnt finished

• Mentor SE-level engineers on SRE craft not just on what to do but on why the learning loop exists: connect the postmortem to the PRR the PRR to the design review the design review to the runbook and the runbook back to the next incident

Developer Experience & Automation (DevX)

• Eliminate toil systematically: identify and automate manual operational work in your domain usingPythonGo orBash. Track toil volume eliminated per quarter time recovered manual touchpoints removed and error modes eliminated through automation

• Build self-service observability tooling that enables development teams to own their own service health dashboards and SLO status pages reducing SRE as a dependency for basic operational visibility

• Track and report DORA metrics Deployment Frequency Lead Time for Change MTTR and Change Failure Rate for your assigned domain on a quarterly basis; use the data to target reliability investments where they will have the most impact

• Partner with development teams to shift reliability practices left: reliability checklists in design reviews SLI instrumentation in definition-of-done runbook templates that developers can author and own

• Contribute to the teams shared SRE tooling library; write reusable well-documented automation modules for common operational patterns

Organizational Influence

This is a core function of the SSE role not a soft skill addendum.

• Drive reliability practice adoption without mandate: you will work with engineering teams that have operated independently for years. Adoption of SLO culture PRR gates and chaos engineering practices depends entirely on your ability to build credibility through demonstrated value not through authority you dont have

• Establish working relationships with engineering domain owners; understand their delivery pressures their quality concerns and what they need from SRE to say yes to new reliability requirements

• Identify the smallest highest-value reliability practice that a skeptical engineering team will adopt first land that win document the outcome and use it to earn the trust needed for the next one

• When reliability practices are not being adopted diagnose the real reason: unclear value unclear ownership too much friction or wrong timing and adapt the approach rather than escalating to mandate

Required Qualifications

• 5 years of experience in SRE DevOps platform engineering or related production-systems roles

• 3 years operating cloud-native distributed systems at production scale with active on-call responsibility

• Demonstrated experience as an on-callIncident Commander (IC)for P1 or P2 incidents structured bridge leadership not just participant involvement

• Experience tuning and validating time-series anomaly detection models in a production observability context this is a Required qualification not a preferred one; anomaly-based detection is a core function of this role

• Strong programming proficiency in at least one ofPythonGo orJavaat production quality capable of writing operational tooling that other engineers will rely on

• Hands-on experience designing SLIs SLOs and managing error budgets for customer-facing or business-critical services

• Deep observability platform experience:PrometheusGrafanaOpenTelemetry and at least one log aggregation solution (Loki Splunk Elasticsearch)

• Organizational influence without authority: demonstrated track record of driving SRE practice adoption in engineering teams that did not initially request SRE involvement this is a Required qualification; technical depth alone is insufficient at this level in a transformation environment

• Fleet-scale deployment awareness: familiarity with progressive rollout strategies blast radius management and configuration drift as a reliability risk in large unattended node deployments

• Strong cloud platform expertise:AWSMicrosoft Azure orGoogle Cloud Platform (GCP)

• AdvancedKubernetesoperational experience: debugging resource management networking policies and workload failure modes. Experience with AI-assisted tooling and development.

Preferred Qualifications

• Experience owning Production Readiness Reviews or service launch gates

• Hands-on chaos or fault injection experience usingLitmusChaosChaos Toolkit orGremlin

• TIC (Technical Incident Commander) certification or equivalent structured incident command training

• Experience operating distributed systems in retail pharmacy healthcare or other operationally sensitive environments where failures have direct patient or customer impact

• LLM integration for operational use cases (alert summarization runbook suggestion incident triage assistance) design or implementation experience

• Experience with streaming data platforms:Apache KafkaRedpandaApache Flink orksqlDB

• Familiarity with analytical databases for observability workloads:ClickHouseApache Druid orTimescaleDB

• Experience with service mesh and traffic management:IstioEnvoyLinkerd

• Infrastructure-as-code proficiency at production scale:TerraformPulumi orAnsible

What Success Looks Like at 6 Months

• You own the SLO health and alerting strategy for your assigned domain with full independence including the ML-based anomaly detection configuration and the false positive rate is measurably lower than when you joined

• You have led at least five P1/P2 incident bridges as Incident Commander with structured postmortems published action items closed andat least two incident classes eliminatedthrough root cause remediation rather than symptom suppression

• You have facilitated your first domain-level PRR and signed off on a service launch

• At least two engineering teams in your domain have adopted a reliability practice an SLO health check a runbook standard or a deployment gate that you introduced without top-down mandate

• You have eliminated measurable toil in your domain: at least three recurring manual tasks automated and documented with before/after comparisons

• The engineering domain owner you partner with describes SRE as a force multiplier for their teams delivery velocity not as a gating function that slows them down

Education:

• Bachelors degree in Computer Science Engineering or a related field or equivalent practical experience