Lead Principal Site Reliability Engineer

Key Responsibilities

Reliability Strategy and Technical Leadership

• Define and drive the site reliability engineering strategy for large-scale distributed and business-critical platforms.
• Establish reliability standards engineering practices and operational readiness requirements across multiple teams.
• Serve as a senior technical authority for system reliability scalability resilience performance and production operations.
• Influence architecture and design decisions to ensure systems are supportable observable fault tolerant and capable of meeting availability objectives.
• Identify systemic reliability risks and lead cross-functional initiatives to address them.
• Provide technical direction and mentorship to site reliability engineers software engineers platform engineers and operations teams.
• Lead technical reviews and promote consistent engineering practices across the organization.

Service Reliability and Observability

• Define and implement service-level indicators service-level objectives error budgets and operational health metrics.
• Develop comprehensive monitoring logging tracing alerting and observability strategies.
• Improve the quality and actionability of alerts while reducing unnecessary operational noise.
• Establish dashboards and reporting mechanisms that clearly communicate service health performance capacity and risk.
• Use production data and reliability trends to prioritize engineering investments and continuous-improvement initiatives.

Automation and Platform Engineering

• Design and implement automation that reduces manual intervention operational toil and human error.
• Build or enhance tools for deployment configuration management infrastructure provisioning incident response and service recovery.
• Promote infrastructure-as-code policy-as-code automated testing and repeatable deployment practices.
• Partner with development teams to improve continuous integration and continuous delivery pipelines.
• Develop self-healing and automated remediation capabilities where appropriate.
• Contribute production-quality software and reusable platform components using modern programming and scripting languages.

Incident Management and Problem Resolution

• Provide technical leadership during complex high-severity production incidents.
• Coordinate diagnosis containment recovery and stakeholder communication during service disruptions.
• Lead blameless post-incident reviews and ensure that corrective actions address root causes rather than symptoms.
• Identify recurring failure patterns and develop long-term engineering solutions.
• Improve incident-management processes escalation procedures runbooks and recovery playbooks.
• Participate in an on-call rotation or provide senior escalation support for critical services as required.

Capacity Performance and Resilience

• Lead capacity planning performance analysis load testing and demand forecasting for critical platforms.
• Identify performance bottlenecks and recommend architectural or operational improvements.
• Design and validate high-availability disaster-recovery backup and business-continuity capabilities.
• Lead resilience testing failure-mode analysis game days and controlled fault-injection exercises.
• Ensure recovery-time and recovery-point objectives are defined tested and achievable.

Security and Operational Governance

• Partner with security and compliance teams to embed security into infrastructure automation and operational practices.
• Support vulnerability remediation access-control improvements audit readiness and secure configuration management.
• Ensure production environments meet organizational standards for change management data protection and operational governance.
• Balance reliability security delivery speed cost and business priorities when recommending technical solutions.

Required Qualifications

• Extensive professional experience in site reliability engineering software engineering cloud infrastructure platform engineering systems engineering or a related technical discipline.
• Demonstrated experience designing operating and improving highly available production systems at significant scale.
• Deep knowledge of distributed systems cloud architecture networking operating systems storage databases and service dependencies.
• Advanced experience with at least one major cloud platform such as Oracle Cloud Infrastructure Amazon Web Services Microsoft Azure or Google Cloud Platform.
• Strong experience with containerization and orchestration technologies including Docker and Kubernetes.
• Proven expertise with infrastructure-as-code and configuration-management technologies such as Terraform Ansible Chef Puppet or equivalent tools.
• Experience implementing observability solutions using metrics logs traces dashboards and automated alerting.
• Strong programming or scripting skills in one or more languages such as Python Go Java JavaScript Bash or similar.
• Experience with continuous integration continuous delivery automated testing and modern release-management practices.
• Demonstrated leadership during critical production incidents and complex technical investigations.
• Ability to diagnose difficult system issues across applications infrastructure networks databases and cloud services.
• Strong written and verbal communication skills including the ability to explain technical risk and recommendations to engineering leaders and business stakeholders.
• Proven ability to lead cross-functional technical initiatives without relying solely on formal authority.

Preferred Qualifications

• Experience supporting enterprise-scale cloud services software-as-a-service platforms or other high-availability customer-facing systems.
• Experience defining and operating service-level objectives error budgets and reliability scorecards.
• Knowledge of chaos engineering resilience testing and automated recovery techniques.
• Experience with multi-region hybrid-cloud or multi-cloud architectures.
• Familiarity with security frameworks compliance requirements and regulated operating environments.
• Experience improving cloud cost efficiency capacity utilization or infrastructure performance.
• Contributions to internal engineering standards technical communities open-source projects or industry publications.
• Bachelors or advanced degree in computer science engineering information systems or a related field or equivalent practical experience.