Software Engineer â Annotation & Evaluation (AI Platform)
Software Engineer â Annotation & Evaluation (AI Platform)
Employer Active
Job Alert
You will be updated with latest job alerts via email
Share
Job Alert
You will be updated with latest job alerts via email
Job Description
At Datadog we are scaling our AI platform that enables the safe and effective development of AI features across the whole of Datadog. The Annotation & Evaluation team is a key player in this mission. We build infrastructure to help teams evaluate their models behaviour through labeled datasets automated testing environments and benchmark tracking tools.
Our usersapplied scientists AI engineers and product teamsdepend on our systems to iterate safely and ship models with confidence. We work across the stack to integrate human and automated evaluation workflows and to ensure model results are explainable and trustworthy.
Were looking for a Software Engineer to contribute to our model evaluation infrastructure and support large-scale annotation and benchmarking workflows. This is a great opportunity for someone interested in the intersection of software engineering AI and data quality.
At Datadog we place value in our office culture - the relationships that it builds the creativity it brings to the table and the collaboration of being together. We operate as a hybrid workplace to ensure our employees can create a work-life harmony that best fits them.
What Youll Do:
- Contribute to evaluation infrastructure that enables model comparison performance tracking and trust & safety scoring.
- Build services and pipelines to support annotation workflows benchmarking dashboards and result aggregation.
- Collaborate with scientists and engineers to build and scale tools for evaluating models in controlled and production-like environments.
- Help maintain integrations with data labeling platforms and model evaluation libraries.
- Write clean testable code to ensure reliability of evaluation tooling and data pipelines.
Who You Are:
- You have 3 years of experience as a backend or platform engineer ideally with exposure to ML systems or data engineering.
- Youre comfortable working in Python Go or similar languages and have built or maintained backend services or pipelines.
- Youre excited by the challenge of evaluating ML/LLM models in real-world contexts.
- You understand the importance of data quality safety checks and explainability in AI systems.
- Bonus points: experience with annotation platforms telemetry replay LLM evaluation tools or metrics dashboards.
Datadog values people from all walks of life. We understand not everyone will meet all the above qualifications on day one. Thats okay. If youre passionate about technology and want to grow your skills we encourage you to apply.
Benefits and Growth:
- New hire stock equity (RSUs) and employee stock purchase plan (ESPP)
- Continuous professional development product training and career pathing
- Intradepartmental mentor and buddy program for in-house networking
- An inclusive company culture ability to join our Community Guilds (Datadog employee resource groups)
- Access to Inclusion Talks our internal panel discussions
- Free global mental health benefits for employees and dependents age 6
- Competitive global benefits
Benefits and Growth listed above may vary based on the country of your employment and the nature of your employment with Datadog.
Employment Type
Full Time
