Senior Software Engineer, Private Cloud Compute Inference Routing

Apple is looking for a senior software engineer to build the next generation of inference routing for Private Cloud Compute the backbone of Siri AI and Apple Intelligence. Youll design distributed systems that direct inference requests across a global fleet of inference servers balancing latency availability and compute efficiency while preserving PCCs privacy guarantees. Your work will directly shape how reliably and efficiently Apple delivers cloud inference at join iCloud Platform Engineering team. We build the server stack behind Push Notifications iMessage FaceTime iCloud iCloud Private Relay and Private Cloud Compute. Our internet scale distributed systems deliver real-time experiences with very low latency and very high availability. They reach hundreds of millions of users across a billion devices worldwide and help them connect and share their lives securely and privately. Our work spans identity messaging queueing presence routing relays developer APIs and much more. Were a small highly collaborative team with a large impact.

Routing LLM inference at Apples scale requires systems that adapt to changing demand available capacity and failures while maintaining low latency and high utilization. PCC adds a fundamental architectural requirement: request content remains encrypted outside trusted compute nodes. Youll develop routing and scheduling strategies within this privacy model working with inference and security teams to evaluate techniques such as cache aware request placement and prefill/decode design system architectures specify custom network protocols and write high performance code. Youll also build scaling strategies for services used by hundreds of millions of people. Our team is small and highly visible so your chances to make an impact are wide open. If you take opportunities like that and run with them youll enjoy working own technical decisions from early design through production using benchmarks and operational evidence to evaluate tradeoffs. Youll work closely with model serving infrastructure security and SRE engineers to turn new inference requirements into reliable routing capabilities.

Design build and operate PCCs inference routing and request scheduling services including global traffic management capacity aware request placement load balancing and adaptive routing strategies that respond to changing demand fleet health and available compute capacity while maintaining latency and availability admission control prioritization and overload management mechanisms that keep the service responsive and allocate capacity fairly across competing with model serving and infrastructure teams to evaluate and develop routing strategies for prefill/decode disaggregation and cache locality optimizations within PCCs privacy network protocols and implement high-performance server software for inference request dispatch and response performance metrics and benchmark routing strategies under realistic workloads using queueing delay request latency throughput and fleet utilization to guide projects from architecture through production rollout partnering with Siri AI Apple Intelligence infrastructure security and SRE teams to deliver new capabilities and improve reliability.

3 years of professional experience designing building and operating large distributed systems in productionnServer development in Java C/C C# Go or SwiftnKnowledge of LLM inference serving including inference request routing KV-cache placement and reuse (e.g. prefix-aware routing) prefill/decode disaggregation and the tradeoffs between latency throughput and accelerator usenExperience with network protocols application level networking and client-server architectures including request routing load balancing or scheduling at scalenExperience benchmarking debugging and tuning the performance of distributed systems plus knowledge of Unix system internalsnExperience with cloud deployment and observability technologies such as KubernetesnBachelors degree in an engineering related field or equivalent experience

Experience building LLM inference serving or routing systems such as llm-d NVIDIA Dynamo vLLM SGLang or similarnFamiliarity with KV cache aware routing prefix caching prefill/decode disaggregation and accelerator capacity placementnExperience with TLS X.509 certificate evaluation or other secure protocols and awareness of recent developments in standard protocolsnUnderstanding of cryptographic protocols encryption key management and privacy-preserving system design (e.g. Oblivious HTTP remote attestation)nExperience leading a substantial engineering project from design through production resolving technical tradeoffs and coordinating across to explain technical tradeoffs collaborate across engineering teams and make progress on problems with incomplete requirements.