AI Quality Analyst

This is a remote position.

Reporting to Manager Quality Engineering & AI Validation focuses on validating the quality of AI-generated outputs agent behaviors and AI-assisted workflows. Builds benchmark scenarios defines scoring rubrics evaluates business usefulness and identifies failure patterns that conventional pass or fail software testing would not catch.

Key Responsibilities

AI Output Evaluation

Design and execute structured evaluations for AI-enabled features and workflows.

Assess outputs for groundedness instruction adherence consistency usefulness tone control compliance and risk.

Identify hallucinations unsupported assertions missing logic and unsafe recommendations.

Benchmark & Rubric Development

Build and maintain golden datasets benchmark prompts comparison sets and scorecards.

Develop rubrics that allow quality to be measured consistently across releases and changes.

Workflow & Model Change Validation

Compare performance across prompt versions workflow revisions tools and models.

Support release decisions with evidence on quality regression or improvement.

Business & Domain Partnership

Work closely with Finance SMEs product managers and engineers to determine what acceptable looks like in real business contexts.

Help define human-review thresholds and escalation patterns for higher-risk use cases.

Production Feedback

Analyze reviewer feedback override patterns and live quality signals to improve evaluation coverage over time.

Requirements

Required Qualifications

4 years of experience in QA analytics business process validation AI evaluation operations or similar roles.

Strong writing analysis and pattern-recognition skills.

Experience evaluating outputs against nuanced criteria rather than only binary correctness.

Ability to work with structured rubrics scenario libraries and evidence-based reviews.

Comfort collaborating across Engineering and business teams.

Experience with finance accounting FP&A transaction services or business process design preferred.

Bachelors degree preferred.

You Are

Thoughtful precise and highly discerning.

Strong at spotting subtle output problems others miss.

Comfortable with ambiguity but disciplined in scoring and documentation.

Focused on trust usefulness and business reality.

Benefits