Sr. Observability Engineering

POSITION SUMMARY

McKesson is seeking a Senior Observability Engineer to join our Platform Engineering this role you will serve as a keyengineerand practitioner of McKessons enterprise observability strategysupportingthe design deployment and continuous improvement of our monitoring and telemetry platforms across cloud and on-premises infrastructure. You will operate with a high degree of autonomy applying SRE principles and an engineering-first mindset to ensure the reliability performance and availability of mission-critical healthcare technology systems.

You will influence observability standards across the organization mentor colleagues and drive innovation without direct management responsibilities. Promotions beyond this level are selective and driven by demonstrated business impact.

KEY RESPONSIBILITIES

Observability Platform Ownership

• Engineer implement and operate enterprise-grade observability platforms including DynatraceLogicMonitor Grafana and Prometheus across multi-cloud and hybrid environments.

• Lead the adoption and integration ofOpenTelemetry(OTel) standards for distributed tracing metrics collection and log correlation across engineering teams.

• Manage and optimize CiscoThousandEyesfor network path visibility internet performance monitoring and end-user experience insights.

• Define and enforce observability-as-code practices managing configurations through version-controlled pipelines (e.g. Terraform HelmGitOps).

• Evaluate recommend and pilot emerging observability technologies and methodologies to continuously advance McKessons monitoring capabilities.

SRE & Engineering Excellence

• Apply Site Reliability Engineering (SRE) practices including error budgets toil reduction and chaos engineering to enhance system resilience.

• Define implement and report on Service Level Objectives (SLOs) Service Level Agreements (SLAs) and Key Performance Indicators (KPIs) in partnership with engineering and business stakeholders.

• Develop and maintain alerting frameworks that minimize noise and maximize signal fidelity reducing mean time to detect (MTTD) and mean time to resolve (MTTR).

• Build self-service observability tooling and dashboards that enable engineering teams to own their own reliability metrics.

• Lead blameless post-incident reviews (PIRs) and drive remediation actions to prevent recurrence.

Cloud & Infrastructure Monitoring

• Design comprehensiveobservability strategies for cloud-native workloads across AWS Azure and/or GCP including containers (Kubernetes/EKS/AKS) serverless and microservices architectures.

• Instrument infrastructure application and business-layer telemetry (logs metrics traces) to provide end-to-end visibility.

• Manage network performance monitoring and synthetic testing viaThousandEyesto proactively identify and resolve connectivity and latency issues impacting end users.

• Partner with DevOps NetOps and Security teams to integrate observability signals into CI/CD pipelines ITSM workflows and SOC operations.

AI & LLM Observability (Emerging Capability)

• Extend the organizations existing observability platforms to instrument AI-powered applications and autonomous agent workflows capturing the telemetry unique to non-deterministic systems.

• ImplementOpenTelemetryGenAI semantic conventions to standardize how AI agent telemetry (traces spans token usage tool invocations) is collected across frameworks and services.

• Design and maintain tracing pipelines that capture the full agent execution chain from user intent and planner decisions through tool calls retrieval steps and model responses enabling root cause analysis beyond traditional request/response tracing.

• Define AI-specific SLOs and cost budgets covering token consumption hallucination rate tool invocation success rate and agent task completion rate.

• Build dashboards and alerting for AI behavioral signals: model drift performance degradation prompt/context version changes and guardrail violations.

• Collaborate with data scienceMLOps and application teams to integrate observability signals into AI evaluation pipelines and continuous improvement feedback loops.

• Implement security-focused AI monitoring including prompt injection detection PII exposure in logs and anomalous tool call patterns in alignment with OWASP Top 10 for LLMs.

Collaboration & Technical Leadership

• Act as an observability subject-matter expert and trusted resource for engineers architects and product teams across McKesson.

• Develop and maintain internal observability standards runbooks and best-practice documentation.

• Mentor and guide junior and mid-level engineers on observability tooling instrumentation techniques and SRE principles.

• Anticipate organizational scaling needs and proactively direct engineering efforts to stay ahead of capacity reliability and visibility challenges.

• Contribute to the development of new frameworks methodologies and tooling that advance McKessons engineering maturity.

REQUIRED QUALIFICATIONS

• Bachelors degree in Computer Science Information Systems Engineering or a related field or equivalent practical experience.

• 7 years of experience in observability monitoring SRE or platform engineering roles within complex enterprise-scale environments.

• Hands-on expertise with two or more of the following platforms: DynatraceLogicMonitorGrafanaThousandEyesandPrometheus.

• Demonstrated experience implementingOpenTelemetry(OTel) for distributed tracing and metrics instrumentation in polyglot service environments.

• Proficiency with CiscoThousandEyesor similar platformsfor network monitoring synthetic testing and internet intelligence.

• Deep understanding of cloud infrastructure monitoring across AWS Azure and/or GCP including IaaS PaaS and container platforms.

• Strong working knowledge of SRE principles: SLOs SLAs error budgets alerting philosophy and incident management.

• Scripting/automation proficiency in Python Go Bash or equivalent with experience building or extending monitoring integrations.

• Experience with infrastructure-as-code andGitOpspractices (Terraform Helm Ansible etc.) as applied to observability configuration management.

• Demonstrated ability to work independently set direction and deliver results in a complex matrixed organization.

PREFERRED QUALIFICATIONS

• Experience in healthcare IT regulated industries or large-scale enterprise environments.

• Familiarity with AIOps platforms and ML-based anomaly detection capabilities within Dynatrace or similar tooling.

• Knowledge ofeBPF-based observability approaches and agent-less instrumentation techniques.

• Experience integrating observability data into ITSM platforms (ServiceNow PagerDuty) and security workflows (SIEM).