AI Test Manager
Share
Job Location:
Greenwood Village, CO - USA
Monthly Salary:
Not Disclosed
Posted on:
3 hours ago
Vacancies:
1 Vacancy
Job Summary
Job Overview: We are seeking a highly technical AI Solutions Evaluation & Experience Manager to architect the framework that determines the real-world intelligence and accuracy of our AI and analytic products and perform ongoing solution testing and this role you will move beyond traditional QA to implement LLM-as-a-Judge and SLM patterns to evaluate performance at scale.
You will be responsible for managing a team that will calculate and interpret critical technical metrics such as RAGas scores (Faithfulness Answer Relevance) Perplexity and Hallucination rates to determine model accuracy. Additionally you will analyze adoption and utilization metrics to assess the health of our product portfolio and trigger safe product retirements when appropriate.
5 years of experience in management and 3 years of experience in technology or AI are required. A thorough understanding of test script development requirements traceability and testing execution for technical solutions is required.
