Staff Product Manager (Evals)
Share
Job Location:
Palo Alto, CA - USA
Monthly Salary:
Not Disclosed
Posted on:
2 days ago
Vacancies:
1 Vacancy
Job Summary
About Workato
Workato transforms technology complexity into business opportunity. As the leader in enterprise orchestration Workato helps businesses globally streamline operations by connecting data processes applications and experiences. Its AI-powered platform enables teams to navigate complex workflows in real-time driving efficiency and agility.
Trusted by a community of 400000 global customers Workato empowers organizations of every size to unlock new value and lead in todays fast-changing world. Learn how Workato helps businesses of all sizes achieve more at .
Why join us
Ultimately Workato believes in fostering a flexible trust-oriented culture that empowers everyone to take full ownership of their roles. We are driven by innovation and looking for team players who want to actively build our company.
But we also believe in balancing productivity with self-care. Thats why we offer all of our employees a vibrant and dynamic work environment along with a multitude of benefits they can enjoy inside and outside of their work lives.
If this sounds right up your alley please submit an application. We look forward to getting to know you!
Also feel free to check out why:
Business Insider named us an enterprise startup to bet your career on
Forbes Cloud 100 recognized us as one of the top 100 private cloud companies in the world
Quartz ranked us the #1 best company for remote workers
Responsibilities
Were looking for a Staff Product Manager to own evaluations for AI agents at Workato both the internal framework that helps our teams ship better AI features and the customer-facing tools that let builders assess and improve the agents they is a role with a dual mandate. Internally youll establish how Workato evaluates agent quality starting with Agent Studio and expanding to other teams shipping AI capabilities. Externally youll build the evaluation experience that helps business technologists understand why their agents succeed or fail and what to do about right person for this role has actually written evals. Youve built test suites designed evaluation criteria and debugged agent failures in the trenches. You know the gap between eval theory and eval reality and you can translate that practitioner knowledge into products that work for both technical teams and non-technical builders.
In this role you will also be responsible to:
Define and own the evaluation framework for Workatos internal AI agent features driving adoption across teams starting with Agent Studio
Build the customer-facing evaluation experience how builders test measure and improve agents they create on Workato
Make hard calls about what evaluation complexity to expose versus abstract balancing rigor with approachability
Partner closely with the Build Experience PM to ensure evaluation is integrated into the builder journey not bolted on
Work with ML engineers and platform teams to ground the framework in technical reality while keeping it accessible
Establish metrics for what good looks like both for internal agent quality and for customer evaluation adoption
Spend significant time with customers understanding where they struggle to assess agent performance and what mental models they bring
Requirements
Qualifications / Experience
7 years in Product Management
Hands-on experience writing evaluations for AI/ML systems (agents LLMs or similar)
Track record of shipping technical products to both internal and external users
Experience driving adoption of frameworks or practices across engineering teams
Strong written and verbal communication skills
Bachelors degree or equivalent experience
Practitioner depth in evaluations. Youve written evals yourself built test suites designed rubrics debugged why agents underperformed. You understand evaluation methodology not only from reading about it but from doing it. You have opinions about what works what doesnt and where current approaches fall short.
Strong product management experience. Youve shipped products driven roadmaps and led cross-functional teams. You know how to translate technical capabilities into user value and write specs that dont leave details to chance.
Technical translation ability. You can take complex evaluation concepts and make them accessible to business technologists without dumbing them down. You understand the difference between hiding complexity and organizing it.
Internal influence skills. Youve driven adoption of frameworks practices or tools across teams. You can be a credible partner to ML engineers while advocating for what internal teams actually need.
Greenfield comfort. Youve defined products from ambiguity scoped v1s made bets with incomplete information and iterated based on what you learned. You dont need an existing playbook to be effective.
B2B product sensibility. You see enterprise conventions as problems to solve not constraints to accept. Youre drawn to products that make complex workflows feel elegant.
Nice to Have
Experience with agent architectures RAG systems or LLM application development
Background in ML engineering solutions architecture or technical program management before PM
Experience building developer tools or platform products
Familiarity with evaluation frameworks (e.g. human eval pipelines automated benchmarks red-teaming)
(REQ ID: 2538)
Required Experience:
Staff IC
Key Skills
- Time Management
- Data Analytics
- Analytical
- Agile
- Requirement Gathering
- Strategic thinking
- Visio
- Communication
- Problem Solving
- Market Research
- UML
- Cross Functional Teams
About Company
A single platform to orchestrate data integration, app connectivity, and process automation across your organization.
