Senior Product Manager I, AI Agent Quality

Role Summary

Our mission at HubSpot is to help millions of organizations grow better. As a Senior Product Manager on the AI Agent Quality team youll lead strategy and execution for one of HubSpots most critical AI investments. Youll shape how customers measure improve and scale the performance of AI agentshelping them confidently use AI to transform marketing sales and service. If youre excited by cutting-edge machine learning and eager to turn innovation into real customer impact wed love to meet you.

In this role youll get to

• Own the Agent Quality strategy across operational product and customer-facing layers defining what good means for reliability ROI and safety.
• Implement evaluation frameworks including golden datasets canary releases with rollback and automated quality gates for model updates.
• Define and standardize metrics like acceptance criteria error budgets SLOs and latency targets to ensure scalable trustworthy AI experiences.
• Deliver customer-facing tools in Breeze Studio that enable managers to inspect interactions trace tool actions and continuously improve agent behavior.
• Mature the Agent Quality Dashboard to unify ROI usage cost and quality data for clear insights and decision-making.
• Build self-serve evaluation kitsscenario libraries regression suites and human-in-the-loop review flowsto empower teams to improve agents safely.
• Instrument end-to-end telemetry and define KPIs such as regression rate time-to-detect safety incidents and cost per successful task.
• Partner with MLOps and AI Foundations teams to ensure scalable performant and cost-efficient operations.
• Establish an incident management program with runbooks and post-mortems that raise the bar for reliability and customer trust.

We are looking for people who have

Required Qualifications

• Proven experience using AI/ML technologies (GenAI NLP Deep Learning) to deliver customer-facing products.
• Track record of leading cross-functional teams and driving measurable outcomes in a fast-moving environment.
• Strong business acumen with data-driven decision-making and KPI ownership.
• Ability to simplify complex ideas and communicate clearly across technical and non-technical audiences.
• Comfort navigating ambiguity and defining clear strategies where none exist.
• Experience collaborating across product design engineering and research disciplines.
• Passion for AI experimentation and delivering meaningful reliable innovation for customers.

Nice-to-Have Qualifications

• Experience building or managing agentic AI systems or evaluation pipelines.
• Familiarity with large-scale telemetry observability or reliability programs.
• Exposure to MLOps prompt engineering or human-in-the-loop feedback systems.
• Previous experience in SaaS product management or AI-powered customer experience tools.