AI Operations Analyst I

We are hiring an AI Operations Analyst (Human-in-the-Loop Strategy & Operations) to help scale and mature HITL practices across Customer Support and Internal Operations.

This role focuses on designing operationalizing and improving HITL processes that support reliable measurable AI agent improvement over time.

You will have ownership of defined HITL processes metrics and outcomes and will work closely with Initiative Leaders HITL Leads AI Product Engineering Operations and Governance teams to extend those practices across initiatives.

HITL Program Enablement

• Own defined components of HITL processes (e.g. review cadence coverage standards tracking mechanisms escalation paths) and support their adoption across initiatives.
• Translate HITL requirements into repeatable operational workflows used by initiative teams.
• Monitor execution and follow-through to ensure consistency quality and completion.

AI Quality & Improvement

• Drive AI agent improvement through structured closed-loop processes:
• Review outcomes root cause analysis corrective actions validation.
• Ensure reviews are completed using real production transactions and focus on the highest-impact issues.
• Identify patterns across review data and coordinate resolution with initiative teams.

Analytics & Reporting

• Define and maintain HITL metrics and performance indicators.
• Analyze trends across initiatives to identify improvement opportunities and emerging risks.
• Partner with BI/Data teams to shape dashboards and recurring reporting for visibility into HITL activity and outcomes.

Cross-Functional Collaboration

• Work with Initiative Leaders HITL Leads Product Engineering and Operations to align priorities and support execution.
• Surface risks gaps and dependencies and escalate when progress is blocked.
• Provide input on HITL effectiveness and readiness based on observed outcomes.

Governance & Risk Support

• Align HITL processes with AI governance and quality requirements.
• Help translate governance expectations into operational review and validation workflows.
• Identify issues that may introduce risk as AI agents scale in usage or complexity.

Scaling & Process Maturity

• Identify opportunities to standardize or improve HITL workflows as adoption increases.
• Contribute to documentation playbooks and shared practices to support consistency across initiatives.
• Support pilots of new HITL tools evaluation approaches or process changes.

What Success Looks Like

• HITL processes are applied consistently across initiatives.
• AI agents show measurable improvement over time.
• Review insights lead to implemented and validated changes.
• Leaders have visibility into HITL execution trends and risks.
• HITL practices continue to function effectively as scale increases.

What Were Looking For

• 4-6 years experience in program operations quality operations support operations product ops trust & safety ops or AI/ML operations (or equivalent).
• Experience working across multiple teams or initiatives in complex operational environments.
• Ability to analyze trends identify root causes and prioritize issues.
• Strong collaboration and communication skills.
• Comfort working in evolving environments where processes are still being defined.

Preferred Experience

• Experience with Customer Support QA contact center/case workflows or internal operations quality programs.
• Familiarity with HITL programs evaluation rubrics