Enter a job title or keyword
Agent Quality Evals Engineer 1754
Job Location:
Monthly Salary:
Experience Required:
Posted:
24 June 2026 (30+ days ago)
Posted:
Application Deadline:
21 September 2026
Application Deadline:
Vacancies:
1 Vacancy
Vacancies:
Job Summary
This is a remote position.
Owns the eval harness and quality gate from the beginning. This role replaces the old late-stage Evals Specialist model with a standing owner for measurable agent quality.
Key Responsibilities
Build and maintain the MVP eval harness: golden tasks exception tasks scorecard metrics and regression packs.
Wire evals into CI so quality regressions fail builds and releases.
Define and maintain release-gate thresholds with Product and the Tech Lead.
Lay the path for later adversarial and drift-testing expansion without overbuilding MVP scope.
Requirements
Must-Have Qualifications
Experience evaluating ML LLM or non-deterministic systems.
Strong test and benchmark design capability.
Comfort working with noisy metrics thresholds and probabilistic behavior.
Good scripting and automation skills.
AI-First Expectations
Uses AI to generate candidate eval cases and failure hypotheses but never confuses generated tests with validated quality.
Approaches AI quality as an operating system not a QA afterthought.
What Success Looks Like in the First 90 Days
The first reference agent has a published scorecard and gated eval path. Golden and exception tests run automatically. The team can explain what good enough to ship means in measurable terms.
Required Skills:
Experience evaluating ML LLM or non-deterministic systems. Strong test and benchmark design capability. Comfort working with noisy metrics thresholds and probabilistic behavior. Good scripting and automation skills. AI-First Expectations Uses AI to generate candidate eval cases and failure hypotheses but never confuses generated tests with validated quality. Approaches AI quality as an operating system not a QA afterthought.
