AI Quality and Evaluation, Lead

About the role

QADs products increasingly include AI agents that make and execute recommendations inside customers operations. These systems behave differently from traditional software outputs vary quality is judgment-based rather than binary and the cost of getting it wrong matters. Were building the function that makes AI quality measurable defensible and continuously improved across our portfolio and were hiring the person to own it.

Youll design and run the evaluation framework our AI products are measured against define the criteria that gate every release and monitor production performance so quality issues are caught early rather than after customers feel them. The role partners closely with product engineering and customer-facing teams and reports to the Head of Product Operations.

If youve worked on the quality and measurement side of LLM-based products and want a role where evaluation is genuinely load-bearing rather than an afterthought this is that role.

What youll own

• Build and own a shared evaluation framework across the product organization: golden datasets LLM-judge rubrics and code-based checks. Measure not just response quality but the quality of the decisions the AI products produce did the recommendation actually serve the customer outcome the product was built for

• Own the technical criteria for stage-gate release reviews: define what evaluation evidence a product must produce to clear each gate and each capability-tier progression. You dont chair the gates; the Head does. But a gate cannot pass without your evidence.

• Own decision auditability: every AI-driven recommendation must be logged with the context considered the rationale and the outcome in a way thats faithful end-to-end and useful for both customer trust and continuous improvement of the product.

• Own production drift detection: monitor evaluation-score regression human-override-rate increase and exception-rate spikes. Treat these as leading indicators of customer issues and surface findings before they show up as support escalations. When a product regresses you trigger a capability-tier review.

• Define blast-radius and rollback requirements with engineering and gate releases on thresholds in CI.

• Partner with each product teams AI lead to translate what good looks like into rubrics breaking quality into independent dimensions (correctness constraint compliance decision quality latency tone) rather than one blended score.

• Run a regular evaluation-review cadence across teams surface regressions early and build the organizations shared vocabulary for what product quality means in this domain.

Qualifications :

What were looking for

• 5-8 years in product ops AI/ML product data or quality engineering with hands-on AI evaluation experience on production LLM systems.

• Strong fluency with agent architecture: traces and spans tool calls RAG/groundedness LLM-as-judge offline vs. online evaluations drift monitoring.

• Practical technical skills SQL plus Python or JavaScript; comfort with evaluation/observability platforms and Git.

• A discovery mindset toward evaluation: failure data and edge cases are the core of the job not a postmortem activity.

• Bias toward decision quality over surface quality. We are not optimizing for hallucination rate alone were measuring whether the AIs recommendation actually served the customers need.

Additional Information :

• Your health and well being are important to us at QAD. We provide programs that help you strike a healthy work-life balance.
• Opportunity to join a growing business launching into its next phase of expansion and transformation.
• Collaborative culture of smart and hard-working people who support one another to get the job done.
• An atmosphere of growth and opportunity where idea-sharing is always prioritized over level or hierarchy.
• Compensation packages based on experience and desired skill set

What success looks like in 6 months

• The shared evaluation framework is in production for the first AI-powered product team. At least one product has measurable decision-quality scores running on a real golden dataset with rubrics that distinguish correctness from constraint compliance from decision quality.

• Decision auditability is verified end-to-end for at least one product in pilot. Verified means you can pull a specific past decision trace the inputs and rationale and demonstrate that data is captured in a way that supports continuous improvement.

• Stage-gate criteria are written calibrated and used. The Head has chaired gate reviews using the evidence specs you produced. At least one gate has been held on evaluation evidence not on schedule pressure and you defended that hold.

• A regular evaluation-review rhythm is running with the product team surfacing at least one regression that was caught before users felt it.

• Production drift monitoring is live on at least one shipped product from the existing portfolio not just on new builds. Coverage spans the full product portfolio not just new work.

What success looks like in 12 months

• The evaluation framework covers every shipped AI-powered product in the portfolio. Coverage is the bar; uneven coverage is the failure mode.

• At least one product has been promoted up the capability ladder on the strength of your evidence. The promotion was defensible to a regulator-style audit.

• Production learning has produced demonstrable platform improvement. You can point to specific patterns captured in production that now ship with subsequent products reducing time-to-deployment for the next product team.

• The evaluation discipline is propagating outward. Product AI leads are writing their own initial rubrics with you reviewing rather than authoring evaluation-writing is becoming a shared skill not your bottleneck.

• At least one regression you caught early prevented a customer incident that would otherwise have surfaced as a support escalation or a usage drop. The ROI of the function is named and known.

About QAD:

QAD  Redzone is redefining manufacturing and supply chains through its intelligent adaptive platform that connects people processes and data into a single System of Action. With three core pillars Redzone (frontline empowerment) Adaptive Applications (the intelligent backbone) and Champion AI (Agentic AI for manufacturing) QAD Redzone helps manufacturers operate with Champion Pace achieving measurable productivity resilience and growth in just 90 days.

QAD is committed to ensuring that every employee feels they work in an environment that values their contributions respects their unique perspectives and provides opportunities for growth regardless of background. QADs DEI program is driving higher levels of diversity equity and inclusion so that employees can bring their whole self to work.

We are an Equal Opportunity Employer and do not discriminate against any employee or applicant for employment because of race color sex age national origin religion sexual orientation gender identity status as a veteran and basis of disability or any other federal state or local protected class. 

Remote Work :

No

Employment Type :

Full-time