AI Observability engineer
Share
Job Location:
San Francisco, CA - USA
Monthly Salary:
Not Disclosed
Posted on:
6 hours ago
Vacancies:
1 Vacancy
Job Summary
AI Observability engineer
Location: remote contract
AI Observability Engineer Tasks
- Design and implement endtoend observability for AI agents models MCPs and data pipelines
- Instrument agents for traces metrics and logs covering prompts tool calls responses latency errors and cost
- Monitor agent behavior reliability and performance across single and multiagent systems
- Build and operate an evaluation framework (offline continuous) for agentic systems
- Define offline golden test suites regression sets and scenariobased evaluations
- Implement continuous inproduction evaluations to detect quality and safety drift with alerts and thresholds
- Implement AI quality and safety metrics (hallucination rate grounding accuracy tool success rate confidence scores)
- Detect and alert on model drift data drift and concept drift impacting agent outcomes
- Implement HumanintheLoop (HITL) review workflows for approvalgated agent actions
- Enforce and log approvals for sensitive or highrisk tool actions
- Define HITL triggers using confidence thresholds escalation policies and reviewer queues
- Feed human feedback back into prompt updates retrieval tuning and agent policy improvements
- Instrument MCPs for request/response observability and correlate MCP telemetry with agent traces
- Integrate observability and evaluation checks into CI/CD pipelines to enable safe rollout canarying and rollback
- Build dashboards and alerts for agent health quality safety and usage trends
- Ensure security privacy and compliance observability including PII detection and audit logging
- Optimize observability cost and performance across logs metrics traces and evaluation runs
- Experience implementing AI observability using AWS cloud services and opensource tooling
Required Skills:
Artificial Intelligence
Key Skills
- ASP.NET
- Health Education
- Fashion Designing
- Fiber
- Investigation
