Senior Applied Scientist Observability Data Platform
Senior Applied Scientist Observability Data Platform
Employer Active
Job Alert
You will be updated with latest job alerts via email
Share
Job Alert
You will be updated with latest job alerts via email
Job Description
The Observability Data Platform (ODP) powers the core of Datadogs telemetry systems handling exabytes of multimodal observability data. As AI agents become first-class consumers of telemetry ODP is evolving to meet their demands - scaling with explosive data growth exposing new query mechanisms rethinking how telemetry is stored transformed and served and enforcing guardrails that ensure security and reliability.
Our teams new focus is to build an intelligent control plane for production systems. This involves moving beyond passive monitoring to create a platform where AI agents can safely and effectively take action in live environments. To achieve this we are integrating techniques from symbolic reasoning formal methods and generative AI.
We are looking for an experienced Senior Applied Scientist with a background that spans systems engineering AI and formal reasoning. You have expertise in areas like causal modeling generative simulation runtime verification or reinforcement learning and are motivated to apply these skills to build reliable systems.
You will join the team behind Datadogs most ambitious projects: evolving observability infrastructure for stochastic self-improving systems.
At Datadog we place value in our office culture - the relationships and collaboration it builds and the creativity it brings to the table. We operate as a hybrid workplace to ensure our Datadogs can create a work-life harmony that best fits them.
What Youll Do:
- Design and prototype intelligent systems for AI-native observability including cost-aware agent orchestration adaptive query execution and self-optimizing system components.
- Apply reinforcement learning search or hybrid approaches to infrastructure-level decision-making such as autoscaling scheduling or load shaping.
- Collaborate with AI researchers and platform engineers to design experimentation loops and verifiers that guide LLM outputs using runtime metrics and formal models.
- Explore emerging paradigms like AI compilers programming after code and runtime-aware prompt engineering to inform Datadogs infrastructure and product design.
- Help define the direction of BitsEvolve - Datadogs optimization agent that uses LLMs and evolutionary search to discover code improvements optimize GPU kernels and tune configurations to improve performance.
- Partner with product teams and platform stakeholders to ensure scientific advances translate into measurable improvements in cost performance and observability depth.
Who You Are:
- You have a BS/MS/PhD in a scientific field or equivalent experience
- You have 8 years of experience in systems engineering database internals or infrastructure research including hands-on experience in a production environment
- You have a strong software engineering foundation ideally in C Rust Go or Python and are comfortable writing performant maintainable code
- You have deep expertise in at least one of the following areas: query optimization data center scheduling compiler design reinforcement learning or distributed systems design
- You have experience applying search planning or learning techniques to solve real-world optimization problems
- You are excited by systems that learn adapt and improve over time using feedback from runtime metrics and human-defined objectives
- You are hypothesis-driven and enjoy designing experiments and evaluation loops whether through simulations benchmarks or live systems
- You thrive in ambiguity enjoy reading papers and building prototypes and want to help shape the future of infrastructure in the AI era
- You enjoy collaborating across research engineering and product to bring scientific insights to practical outcomes
Datadog values people from all walks of life. We understand not everyone will meet all the above qualifications on day one. Thats okay. If youre passionate about technology and want to grow your skills we encourage you to apply.
Benefits and Growth:
- Get to build tools for software engineers just like yourself. And use the tools we build to accelerate our development.
- Have a lot of influence on product direction and impact on the business .
- Work with skilled knowledgeable and kind teammates who are happy to teach and learn
- Competitive global benefits
- Continuous professional development
Benefits and Growth listed above may vary based on the country of your employment and the nature of your employment with Datadog.
Required Experience:
Senior IC
Employment Type
Full Time
