Observability & Evaluation Engineer
Job Location:
Charlotte, NC - USA
Monthly Salary:
Posted:
Application Deadline:
Vacancies:
Job Summary
Job Title: Observability & Evaluation Engineer
Rate: 65-70/ Hr on C2C
Type -- Contract
Location: Charlotte NC (3 days onsite)
Experience: 10 Years
Job Description
We are looking for an experienced Observability & Eval Engineer to build and maintain monitoring tracing and evaluation solutions for AI/LLM applications and agent-based systems. The candidate will ensure AI applications perform reliably in production by implementing telemetry dashboards alerts automated evaluations and service-level objectives.
Key Responsibilities
- Implement monitoring tracing and telemetry for AI/LLM applications and AI agents.
- Develop evaluation frameworks to measure AI response quality accuracy and performance.
- Build dashboards alerts and metrics to monitor application health.
- Analyze prompt and model performance latency errors and reliability.
- Define Service Level Objectives (SLOs) and support production readiness.
- Develop automated tests and evaluation suites using Python.
- Troubleshoot production issues and maintain operational runbooks.
- Collaborate with engineering and AI teams to improve system reliability.
Required Skills
- 10 years of overall IT/software engineering experience.
- Strong Python programming and test automation skills.
- Hands-on experience with LLM/AI agent evaluation.
- Knowledge of observability tracing and telemetry.
- Experience building dashboards alerts and monitoring metrics.
- Understanding of SLOs production operations and incident troubleshooting.
- Familiarity with tools such as OpenTelemetry Grafana Prometheus Datadog LangSmith or Langfuse.
Preferred Qualifications
- Experience supporting GenAI or agentic AI applications in production.
- Knowledge of prompt evaluation model performance analysis and regression testing.
- Experience with cloud platforms and CI/CD pipelines.
Ideal Candidate
A senior engineer with strong Python skills and practical experience in both AI/LLM evaluation and production observability capable of ensuring AI systems are reliable measurable and production-ready.
Disclaimer: Diverse Lynx LLC is an Equal Opportunity Employer. All applicants and employees are evaluated without discrimination based solely on their qualifications ability competence and performance. This email and its attachments may contain confidential or proprietary information and is intended only for the recipient(s). If you received this message in error please disregard it and notify the sender. If you no longer wish to receive our communications you may unsubscribe at any time.
Security Notice: Our official website is We do not operate or authorize any other websites representing Diverse Lynx LLC.
Required Experience:
IC
