Senior AI Engineer CD Visibility

Madrid - Spain

Monthly Salary: Not Disclosed

Posted on: Yesterday

Vacancies: 1 Vacancy

Job Summary

At Datadog we leverage AI across our observability platform to improve monitoring speed up incident resolution and ensure data reliability for cloud applications.

Datadogs Deployment Gates team builds customer-facing systems that decide whether software should ship to production. Deployment Gates sit directly in customers CI/CD pipelines and use observability data to answer one of the hardest questions in software delivery:
Given everything we know right now is this deployment safe to proceed

In this role you will:

Work on the evolution of Deployment Gates from static rules to AI-driven decisions combining Datadog telemetry to detect and block faulty changes before they impact users.
Lead the design of progressive deployment automation starting with zero-setup conservative AI rules and evolving toward adaptive gates that learn from incidents and organizational patterns.
Design the foundation for autonomous remediation connecting what changed in code to what broke in production from blocking and rolling back unsafe deployments to proposing fixes and enforcing policies.

This is a highly productminded engineering role: youll work from problem discovery and UX all the way to reliable scalable production systems.

At Datadog we place value in our office culture - the relationships that it builds the creativity it brings to the table and the collaboration of being together. We operate as a hybrid workplace to ensure our employees can create a work-life harmony that best fits them.

What youll do:

Build AI-driven deployment gates: Design and ship decision systems that evaluate customer deployments using CI/CD context and Datadog telemetry producing safe explainable allow/block outcomes.
Own evals and rollout: Define precision recall and trust metrics; build offline and online evals; validate changes in shadow mode; and safely promote improvements to enforcement.
Design for robustness and safety: Implement conservative defaults guardrails fallbacks and human-in-the-loop paths so gates behave predictably under noisy or incomplete data.
Partner closely with Product: Work hand-in-hand with the Product Manager to translate customer problems adoption signals and roadmap goals into concrete technical decisions and iterations.
Integrate across the Datadog platform: Partner with internal AI teams building the Faulty Deployment Detection pipeline as well as teams working on LLMs and AI agents.
Own production systems: Build and operate reliable backend services that run in the critical path of customer deployments and be on-call for those services.

Who you are:

A Productminded engineer who ships AI to production
You have 5 years experience with backend systems and microservices performance: tracing latency breakdowns concurrency and resiliency patterns
You are proficient in a modern programming language; strong API/service design; production ops (monitoring alerting oncall rotation)
You have proven experience delivering LLM/agent features to production
You are comfortable owning user journeys iterating from prototype alpha GA and measuring impact with clear product metrics
An End-to-end AI implementation owner: You understands the end-to-end LLM product lifecycle
Fluent with offline/online evils for AI systems

Bonus Point:

Experience with Continuous Delivery tools (i.e. ArgoCD / Argo Rollouts Spinnaker Octopus Deploy)
Exposure to planning/agent frameworks tooluse orchestration RAG and retrieval/indexing for observability data

Datadog values people from all walks of life. We understand not everyone will meet all the above qualifications on day one. Thats okay. If youre passionate about technology and want to grow your skills we encourage you to apply.

Benefits and Growth:

New hire stock equity (RSUs) and employee stock purchase plan (ESPP)
Continuous professional development product training and career pathing
Intradepartmental mentor and buddy program for in-house networking
An inclusive company culture ability to join our Community Guilds (Datadog employee resource groups)
Access to Inclusion Talks our Internal panel discussions
Free global mental health benefits for employees and dependents age 6
Competitive global benefits

Benefits and Growth listed above may vary based on the country of your employment and the nature of your employment with Datadog.

Required Experience:

Senior IC

At Datadog we leverage AI across our observability platform to improve monitoring speed up incident resolution and ensure data reliability for cloud applications.Datadogs Deployment Gates team builds customer-facing systems that decide whether software should ship to production. Deployment Gates sit...