Staff Software Engineer - Alerting Platform

We are looking for a Staff Engineer to help us scale Datadogs Alerting Platform which is responsible for the core systems that define and schedule monitors create alerts and ensure the accuracy and timeliness of the end to end alerting process across the platform.

This is a unique opportunity to contribute to one of the most critical platforms at Datadog. Customers can configure monitors and generate alerts for virtually every product in our unified platform. Its imperative that we maintain our customers trust by delivering these notifications reliably. In practice this means the alerting platform has to be the most reliable platform.

As we grow we have to design systems that can degrade furthermore while still ensuring the best customer experience without breaking. This staff engineer will focus on two critical components: the alerting scheduler responsible for scheduling the timely evaluation of millions of monitors each minute and the state processor that makes the critical decision about when a transition in monitor state has occurred. These distributed systems are tied together one being the consumer (state machine) of the other (scheduler). The reliability and fault tolerance of these systems together and across the entire alerting platform is critical to Datadogs customer trust and business success. Upcoming initiatives to achieve an order of magnitude increase in reliability will require deep changes to these complex systems.

At Datadog we place value in our office culture the relationships and collaboration it builds and the creativity it brings to the table. We operate as a hybrid workplace to ensure our Datadogs can create a worklife harmony that best fits them.

What Youll Do:

• Design and drive high priority high visibility projects that increase the platforms resilience and scalability across multiple teams
• Lead and guide others through architectural decisions for new and existing distributed highthroughput realtime systems
• Identify potential system risks and trends in reliability and design solutions to address them
• Provide input on prioritization of engineeringled initiatives in short and longterm planning and roadmaps
• Collaborate closely with partner platforms that integrate and depend on the alerting platform to provide critical capabilities to their customers

Who You Are:

• You have led crossteam initiatives in a platform or infrastructurefocused environment for 2 years.
• You have led impactful technical initiatives in an environment where performance reliability and accuracy are firstorder concerns
• You have a reliabilityoriented mindset and care deeply about designing and building resilient architectures
• You have significant back end programming experienced and have architected built and operated distributed systems to solve problems at high scale

Datadog values people from all walks of life. We understand not everyone will meet all the above qualifications on day one. Thats okay. If youre passionate about technology and want to grow your skills we encourage you to apply.

Benefits and Growth:

• New hire stock equity (RSUs) and employee stock purchase plan ESPP
• Continuous professional development product training and career pathing
• Intradepartmental mentor and buddy program for inhouse networking
• An inclusive company culture ability to join our Community Guilds (Datadog employee resource groups)
• Access to Inclusion Talks our internal panel discussions
• Free global mental health benefits for employees and dependents age 6
• Competitive global benefits

Benefits and Growth listed above may vary based on the country of your employment and the nature of your employment with Datadog.

Required Experience:

Staff IC