Technical Product Manager, Data Science

CENTRL is a leading risk and compliance technology company that provides AI powered enterprise-grade risk due diligence cyber security and privacy management solutions to financial institutions worldwide. Our clients include some of the largest banks and investment management firms across the Americas Europe and APAC. Headquartered in Silicon Valley CENTRL has regional offices in New York India Australia and the United Kingdom. Established in 2015 CENTRL is a high-growth venture backed SaaS firm leading the way in innovative generative AI solutions to manage third party risk and due diligence.

Position Overview:

We are looking for an owner for the accuracy reliability and cost-efficiency of the AI behind CentrlX our agentic platform for Manager Research and Investor Relations teams at investment firms.

We have a strong engineering team building the platform: agents skills knowledge automations connectors and governance. We are looking for a single owner for output accuracy and efficiency.

You will own the model and prompt layer end to end. You will build the evaluation datasets and harnesses that tell us whether a change actually helped run structured comparisons across model providers on accuracy latency and cost and drive the resulting changes into the product. You will help create and design agents and skills and work closely with sales and PS in order to help drive standards for prompting and agents and then feed that back into the product design. You will do a real share of this work yourself from prompt iteration to writing user facing stories like incorporating interactive feedback. While we have support teams to intake client issues and fix them you will own AI Quality and be the point person for what is happening and what needs to improve.

This is a product role not a research role. We want the analytical rigor of a data scientist paired with the judgment of a product manager: someone who can run the experiment interpret it honestly and then turn it into a shipped change.

Key Responsibilities

Evaluation & AI Quality

• Build and own CentrlXs evaluation foundation from the ground up: golden datasets grading rubrics LLM-as-judge pipelines calibrated against human labels and regression suites that run before prompt or model changes ship.
• Define what good means for each core workflow document digitization and extraction retrieval and groundedness Smart Summary Smart Response Smart Evaluation and full multi-step agent runs and set a measurable quality bar for each.
• Evaluate agent behavior not just single responses: tool selection retrieval quality step sequencing and whether the finished deliverable holds up to a practitioners review.
• Turn every real client failure into a permanent eval case so the same class of error does not come back.

Model Selection & Cost Optimization

• Continuously benchmark models across providers OpenAI Anthropic Google open-weight and specialized document models on accuracy latency and cost for each workflow and make the call on what we run where.
• Own model migrations end to end including our in-flight move off GPT-4.1 in document digitization where current alternatives are materially faster cheaper and more accurate.
• Track and manage AI spend by workflow and use routing model tiering caching and context strategy to hold quality while bringing cost down.
• Maintain a working view of the model landscape releases pricing changes deprecations and turn it into a recommendation with evidence attached not a newsletter.

Instrumentation & Data

• Define the logging and tracing we need prompt inputs retrieved context prompt text outputs tool calls token counts latency and write the stories to get it built.
• Partner with Product and Design to build in-app feedback capture (ratings corrections structured reason codes) so labeled data accumulates as a byproduct of normal use instead of a periodic collection project.
• Build and maintain the datasets yourself: pull the data label it curate the hard slices and keep the sets honest with holdouts and rotation.

Ownership & Cross-Functional Partnership

• Serve as the single point of contact for AI quality escalations from Client Success Sales Engineering and Professional Services triage reproduce root-cause and close the loop.
• Write the user stories and acceptance criteria that turn findings into shipped changes and make the prompt configuration and model changes yourself where that is the fastest path.
• Publish a regular quality and cost readout that leadership engineering and client-facing teams all treat as the same version of the truth.
• Work with CENTRLs Manager Research Investor Relations and diligence practitioners to encode domain judgment into rubrics in our market accuracy is defined by industry expertise not by a generic benchmark.

Minimum Qualifications

• Must have work authorization in the USA.
• 3 years across product management data science or applied AI including at least 2 years working on LLM-based products in production.
• Hands-on Python and SQL. You are comfortable in a notebook pulling data running batch inference and computing metrics. This role writes code; it does not only specify it.
• Demonstrated experience building evaluation datasets and harnesses for LLM systems: golden sets rubric design LLM-as-judge with human calibration and regression testing against prompt and model changes.
• Working fluency with at least one eval or LLM observability platform: Braintrust LangSmith Langfuse Arize Phoenix W&B Weave Inspect Promptfoo or a comparable in-house harness.
• Practical understanding of RAG systems: retrieval quality groundedness and faithfulness hallucination detection and chunking and context strategy.
• Statistical literacy: you can size a comparison judge significance and say plainly when a difference is not real.
• Ability to write clear user stories and acceptance criteria and work inside an agile engineering process.
• Strong written communication. This role produces recommendations that executives act on.

Preferred Qualifications

• Experience with document AI: OCR and vision-language extraction table and layout parsing and structured output from complex PDFs.
• Experience evaluating agentic systems multi-step trajectories tool-use correctness and long-run failure modes.
• Experience in financial services or investment management: due diligence manager research investor relations or DDQ/RFP workflows.
• Experience reducing inference cost at scale through routing tiering batching caching or context compression.
• Experience designing in-product feedback mechanisms that generate labeled evaluation data.
• Familiarity with agent frameworks and MCP.
• Degree in a quantitative or technical field.