Senior Applied Scientist AI Platform

AI Platform builds the foundations of Datadogs AI efforts. The org is 70 people organised in three pillars: training and serving (GPU clusters distributed training low-level infrastructure) agents (agent harnesses memory systems the internal AI gateway that routes every LLM request at Datadog) and evaluation and experimentation. This role sits in the evaluation and experimentation pillar which owns Datadogs shared annotation and evaluation infrastructure including the evaluation scenario store and the telemetry archival systems used across the Bits org. Together they let an agent travel back in time and query what Datadog looked like at the exact moment an incident happened so scenarios can be replayed and agent performance tracked over time.

Specifically youll be the first applied scientist on GenSim (Generative Simulations) the team that builds the environments Datadogs agents learn in. GenSim doesnt replay sampled telemetry it stands up real fully instrumented applications that talk to Datadog drives them with representative traffic injects controlled failures and records what happens. Because GenSim injected the failure it knows the ground truth. That corpus hundreds of postmortem-derived scenarios and thousands of runnable applications is today the primary source of post-training data for Datadogs own SRE model and the substrate that Bits AI SRE and our other agents are trained and evaluated against.

The team has no applied science support today and is learning post-training data methodology on the fly. Thats the gap this role fills and the open questions are the interesting part. How do you tell whether a generated environment is actually representative of the messy incomplete telemetry real customers run rather than a suspiciously clean one where every monitor exists and every service emits complete logs How do you make injected problems genuinely hard and how do you even measure difficulty How do you define and control the quality of post-training data when correctness representativeness and difficulty pull in different directions How do you evaluate an agent end to end when the trajectory is non-deterministic Creating simulated agent environments for monitoring and SRE work is not well solved in open source or in published research and Datadog is the leading company in this field. If those are the problems you want to spend your time on come build this with us.

• Own the applied science direction for GenSim: set the methodology and the forward-looking technical calls on how simulated environments and post-training data should be built on a team where that decision-making does not exist yet.
• Define measure and raise the quality of post-training data basic correctness representativeness against the real distribution of customer systems and production telemetry and difficulty and make those measures something the team can act on release over release.
• Close the realism gap. Simulated environments today are too clean and the injected problems are not yet hard enough; youll drive the research and the engineering that make them look like real imperfect production systems.
• Build scalable production-grade systems rather than research scripts. The output is not just a dataset it is a system of synthetic environments that must be reliable and invokable inside a training loop.
• Determine how this data is best applied in LLM post-training and in evaluation and own the agent and LLM application evaluation approaches for these environments.
• Work cross-functionally with the engineers and applied scientists on adjacent teams Bits AI SRE the model training effort and the wider evaluation and experimentation pillar so that what you learn moves freely in both directions.

• You have a PhD MS or equivalent research experience in a scientific field with strong applied mathematics grounding.
• 6 years of relevant applied science or ML engineering experience including setting technical direction for others.
• You have hands-on experience with LLM and agent post-training data: how it is created managed and how training-data quality is controlled. This is the requirement that matters most.
• You have real domain expertise in LLMs and agentic applications not classical ML fine-tuning. Fine-tuning classifiers or traditional models is a different problem from the one this team is solving.
• You have evaluated agents or LLM applications and can define what good means before you measure it.
• You are a strong programmer and production software engineer. Python at minimum plus the ability to ship scalable production systems and work with distributed systems.
• You collaborate well across engineering and science teams and youre comfortable being the domain expert who decides what comes next.
• You thrive in ambiguity and can make sound technical calls when the path isnt yet defined.

• Hands-on LLM fine-tuning post-training or model training experience.
• Background in statistics experiment design and data analysis.
• Experience deploying production-level ML infrastructure.
• Observability or monitoring systems background.
• Architecture-level understanding of LLMs.

• New hire stock equity (RSUs) and employee stock purchase plan (ESPP)
• Continuous professional development product training and career pathing
• Intra-departmental mentor and buddy program for in-house networking
• An inclusive company culture and the ability to join our Community Guilds
• Access to Inclusion Talks our internal panel discussions
• Free global Spring Health benefits for employees and dependents age 6
• Competitive global benefits

Benefits and Growth listed above may vary based on the country of your employment and the nature of your employment with Datadog.

#LI-Hybrid