AI Quality Infrastructure Engineer

Mountain View, CA - USA

Monthly Salary: Not Disclosed

Posted on: 16 hours ago

Vacancies: 1 Vacancy

Job Summary

Job Role: AI Quality Infrastructure Engineer

Job Location: MTV CA or San Diego CA or NYC NY or Remote

Job Duration: Long Term Contract

Overview:

As an AI Quality Infrastructure Engineer you will build quality infrastructure and build quality pipelines with these to guarantee the reliability of our AI ecosystem. You wont just monitor models; you will build the automated systems that make monitoring possible at scale. You will be responsible for engineering the LLM-as-a-judge services custom observability frameworks and the automated alerting logic that connects our AI agents to our production response teams. Your work will bridge the gap between AI research and production-grade reliability engineering.

What the Job Entails

Build Automated Quality Tooling for AI: Build and maintain internal tools and services that automate the measurement of quality for the AI and AI agent development lifecycle. This includes the development of quality coverage tools for prompt-based approaches creating testing automation pipelines to support tool call validations and the LLM-as-a-judge scoring engines.
Build Production Monitoring Tooling: Build and maintain internal tools the ensure continuous production monitoring with synthetic test for our AI capabilities.
Design Synthetic Data Generators: Build tools to programmatically generate high-fidelity synthetic datasets for continuous stress-testing and golden set benchmarking.
Labeling tools: Build and maintain rapid labeling pipelines for AI Agents.
Engineer Observability Pipelines: Develop the backend data pipelines that stream model logs tool-calling traces and metadata into Splunk and Amplitude for real-time visualization.
Automate Alerting & Incident Response: Write the logic and scripts to programmatically trigger PagerDuty incidents based on complex model performance thresholds and data quality anomalies.
Develop Data Quality Services: Create automated services to detect data drift and non-natural language patterns (e.g. input feature distributions or sentiment shifts) before they impact the user.
Scalable ML Tooling: Eventually design and build the infrastructure for end-to-end Machine Learning pipelines focusing on automated training data validation and model-check gatekeeping.

Our Ideal Candidate:

Education: Bachelors or Masters Degree in Computer Science Software Engineering or a related technical field.
Engineering Proficiency: Expert-level Python and SQL skills with a focus on building reusable libraries APIs and automation scripts.
Monitoring-as-Code: Experience in Monitoring-as-Code including programmatically configuring Splunk alerts Amplitude and PagerDuty services.
AI/ML Infrastructure: Strong understanding of LLM architectures and the engineering challenges of testing non-deterministic systems.
System Design Mindset: Ability to design scalable fault-tolerant systems that can handle millions of AI conversation traces without latency.
Problem Solving: A builder mentality-you see a manual process and your first instinct is to write code to automate.

Thanks & Regards

Akhil

Job Role: AI Quality Infrastructure Engineer Job Location: MTV CA or San Diego CA or NYC NY or Remote Job Duration: Long Term Contract Overview: As an AI Quality Infrastructure Engineer you will build quality infrastructure and build quality pipelines with these to guarantee the reliability of ou...