Senior Software Engineer — Service Observability & Detection

Imagine what you could do here. At Apple new ideas have a way of becoming extraordinary products services and customer experiences very quickly. Apple Pay and Wallet are at the heart of how millions of people around the world make payments store credentials and interact with commerce every day. Behind every tap every transaction and every boarding pass is a set of systems that must be fast reliable and secure at massive scale. We are looking for a Senior Software Engineer to join the Service Observability u0026 Detection team within Wallet Payments and Commerce (WPC). In this role you will design and build the telemetry foundations SLO frameworks and detection systems that give engineering and operations teams deep real-time visibility into the health of services that hundreds of millions of people depend on every day. You will define what good looks like for telemetry at scale and build systems that ensure we detect failures before our customers do.

Apple Pay processes billions of transactions across millions of merchants worldwide. The systems behind Apple Pay Apple Wallet Apple Cash Tap to Pay and our commerce platforms must meet the highest standards for availability latency security and correctness with zero tolerance for downtime or data team owns the observability and detection layer across all WPC services. We dont just consume telemetry we define the telemetry strategy build the instrumentation standards and design the SLO and alerting architectures that turn raw signals into actionable detection. We work at the intersection of platform engineering reliability and operations building systems that scale across hundreds of services thousands of APIs and billions of is a senior engineering role within one of Apples most critical and high-scale service organizations. You will work cross-functionally with multiple product and engineering teams partnering directly with offering owners platform teams and operations to build observability and detection systems that the entire organization depends on. You will lead by influence drive adoption across teams you dont manage and set the telemetry and reliability standards that shape how WPC builds monitors and operates services at scale.

Design and build scalable telemetry pipelines using OpenTelemetry including SDK instrumentation libraries Collector deployments (agent and gateway tiers) processors filters exporters and metadata enrichment optimized for high-throughput low-latency environments where every dropped span or miscounted metric has real-world and enforce telemetry standards across a large service organization metric naming conventions span attribute schemas context propagation patterns and cardinality management strategies that keep telemetry useful and cost-effective as the system SLO frameworks from the ground up designing multi-tier SLO models (service-level feature-level journey-level) with proper SLI definitions error budget policies and burn-rate alerting that reflects actual customer experience rather than synthetic intelligent alerting systems that scale multi-layered alert architectures that prevent alert storms during cascading failures route signals to the right responders and provide clear context on customer impact blast radius and root cause and implement end-to-end user journey monitoring across distributed microservice architectures correlating traces metrics and logs across API gateways orchestration layers partner integrations and asynchronous event pipelines to provide flow-level and maintain observability-as-code infrastructure managing monitors SLOs dashboards and service catalog definitions through version-controlled templatized and repeatable deployment tooling and automation that accelerates observability adoption instrumentation gap detection automated service catalog enrichment telemetry health checks and self-service onboarding for engineering hard telemetry engineering problems at scale sampling strategies that preserve signal fidelity tail-based sampling for error and latency outliers high-cardinality metric management cross-service trace stitching and telemetry pipeline reliability under observability coverage across third-party and partner integrations where you dont own the code designing proxy metrics synthetic probes and inferred service detection to eliminate blind spots at system boundaries.

8 years of experience in software engineering with deep expertise in observability reliability engineering or telemetry platform track record of leading cross-functional technical initiatives driving alignment adoption and delivery across multiple engineering teams you dont directly building telemetry and observability systems from the ground up not just configuring tools but designing instrumentation strategies telemetry pipelines and data models that scale to thousands of services and billions of hands-on experience with OpenTelemetry SDK instrumentation Collector architecture (agent and gateway tiers) processors exporters context propagation and metadata enrichment at understanding of SLOs SLIs and error budgets with hands-on experience defining implementing and operating SLO-based alerting in designing alerting architectures for complex distributed systems including burn-rate alerts multi-signal correlation intelligent routing and alert storm proficiency in at least two of: Kotlin Java Go or with observability platforms such as Datadog Grafana Prometheus or Splunk including APM metrics traces monitors and understanding of distributed systems concepts including microservices architectures API gateways event-driven systems and service-to-service solving hard telemetry problems at scale sampling strategies high-cardinality metric management tail-based sampling cross-service trace stitching and pipeline reliability under communication and stakeholder management skills ability to translate complex technical observability concepts into clear strategies for engineering leadership and partner ability to establish org-wide engineering standards and drive adoption through influence documentation and tooling rather than or MS in Computer Science Software Engineering or a related field or equivalent professional experience.

Experience working on payment systems financial services platforms or other high-compliance high-availability building observability platforms or telemetry infrastructure that serves an entire engineering organization not just a single leading technical initiatives across organizational boundaries building consensus defining standards and driving adoption without direct applying AI/ML to operational problems such as anomaly detection intelligent alert grouping automated incident correlation or predictive with incident management workflows and tools understanding how detection feeds into triage escalation and with Infrastructure-as-Code tools such as Terraform or Pulumi for managing observability resources (monitors SLOs dashboards) as with service catalog or service registry systems ownership models dependency mapping and health scoring at organizational building systems that serve both engineering teams and SOC/NOC operators balancing depth of signal with clarity of action.