Enter a job title or keyword
Senior Data Scientist — AI Evaluation & Quality | Internal AI Agents (Remote in Europe)
Job Location:
Monthly Salary:
Posted:
29 September 2026 (10 days ago)
Posted:
Application Deadline:
3 January 2027
Application Deadline:
Vacancies:
1 Vacancy
Vacancies:
Job Summary
About Finom
Finom is a European tech startup headquartered in Amsterdam and were on a journey towards revolutionizing the financial landscape for entrepreneurs worldwide. Our mission is to develop an all-in-one financial B2B solution that integrates banking functions accounting financial management and invoicing into a seamless mobile-first platform.
We recently closed a 115 million Series C equity round (around $133 million) bringing our total funding to approximately $346 million. This significant investment follows a $105 million growth funding round from General Catalyst a long-term backer since 2021 known for supporting companies like Airbnb HubSpot KAYAK and Stripe.
Finoms platform goes beyond traditional banking offering invoicing and a growing suite of features including AI-enabled accounting aiming to simplify financial management for entrepreneurs. Were actively expanding our reach across key EU markets like Germany France the Netherlands Italy and Spain.
At Finom were not just redefining the entrepreneurial experience were empowering our employees to make a real difference. Your work matters and your impact extends far beyond product metrics. We nurture innovation and an inspiring work environment where bold ideas thrive prioritizing thorough research swift implementation of solutions and ensuring that every effort we make benefits our users employees partners and our business as a whole.
Maintaining our start-up spirit we prioritize thorough research swift implementation of solutions and ensuring that every effort we make benefits our users employees partners and of course our business.
Youll join Finoms AI Team as the founding IC dedicated to the quality evaluation and telemetry of AI agents powering Finoms internal operations and tools (including Ops workflows and internal AI analytics engines across 20 core processes).
Our belief: an AI agent is only as good as the evaluation loop running on it. Because internal operational agents directly touch financial compliance and support workflows evaluation is an exact statistical and engineering discipline here.
Your mission: design the evaluation methodology build golden benchmarks and establish quality gates for our internal AI agents from scratchworking directly with process owners and domain experts with no senior quality owner above you to lean on.
Core Stack: Databricks DeepEval Claude Code Cursor Python SQL dbt.
What You Will Be Doing
- Own and extend our offline/online evaluation suites across 20 internal AI agent processesdatasets (capability regression) LLM-as-a-judge rubrics and deterministic checks.
- Establish pre-launch quality gates: enforce pass/fail thresholds in CI/CD pipelines before agent prompt context or tool changes hit production.
- Work directly with domain experts to label cases and resolve annotator disagreementfixing definition criteria rather than averaging disagreement away.
- Build test datasets derived from real user & operational traffic (tickets internal chats colleague queries) rather than synthetic edge cases.
- Harden statistical methodology: handle judge drift verbosity bias non-determinism and measure true metric shifts vs. noise.
- Translate quality numbers into operational decisions: run weekly syncs with process owners to define clear quality vs. cost/latency trade-offs.
Must-Haves
- 5 years in Data Science / Product Analytics / Applied AI roles with sustained product-level metric ownership.
- Production LLM Experience: In the last 12 years you have built shipped or evaluated LLM-based systems (RAG multi-step tool use agents) as a core primary job responsibility.
- Autonomous Quality Ownership: Proven track record of owning evaluation methodology or analytics for an entire product or end-to-end process (what to build vs. what NOT to build).
- Fluent Python & SQL: Ability to write clean data pipelines evaluation harnesses and dbt transformation models directly.
- Statistical Rigor: Applied knowledge of sampling hypothesis testing variance analysis and confidence intervals on noisy metrics.
Daily Setup & Tooling
- AI-assisted coding (Claude Code Cursor or Codex) is your default daily authoring environment for Python SQL and evaluation scriptsnot something you occasionally experiment with.
- You can walk us through concrete work tasks from the last month where AI coding tools accelerated your engineering and data analysis.
How we work one thing we mean seriously
- AI-assisted coding is our default authoring environment not a bonus
- Claude Code is our main tool youll reach for it for SQL Python analyses dashboards and internal scripts
- Were looking for analysts who are already curious and fluent with AI coding or genuinely excited to become fluent fast
- We care about what you ship and how clearly you think
- If this idea excites you rather than worries you youll feel at home here
What You Will Get In Return
Make a genuine impact on the product
Join our upward trajectory and grow with us. We provide the resources and opportunities for continuous personal and professional development empowering you to make a genuine impact on our evolving product.
Work in the EU
Embark on this exciting journey with us and enjoy the flexibility of traveling and working remotely or in a hybrid model across Europe.
Become a stock options holder
Unlock your inner entrepreneur and align your aspirations with ours through our Stock Options Program. This exciting opportunity is available to every team member from junior team members to our founders.
Receive unwavering support and care
Finom stands by you at every step embodying our commitment to your well-being and success reflected in our modern friendly and eco-conscious corporate culture. We offer constant support and care to ensure your Finom experience is successful and fulfilling.
Work & Swim program
Immerse yourself in our exclusive Work & Swim Program. Spend one month in a comfortable corporate apartment in enchanting Cyprus. Its the ideal opportunity to strike the perfect work-life balance while enjoying breathtaking Mediterranean views.
Equal Opportunity Statement
At Finom were an equal opportunity employer and value diversity at our company. We embrace diversity and invite applications from all walks of life. We do not discriminate based on race religion color national origin gender sexual orientation age marital status disability status or other applicable legally protected characteristics.
We may use artificial intelligence (AI) tools to support parts of the hiring process such as reviewing applications analyzing resumes or assessing responses and identifying potential inconsistencies or verification signals in application materials based on available information. These tools assist our recruitment team but do not replace human judgment. Final hiring decisions are ultimately made by humans. If you would like more information about how your data is processed please contact us.
Required Experience:
Senior IC
About Company
What You Will Get In Return Make a genuine impact on the product Join our upward trajectory, and grow with us. We provide the resources and opportunities for continuous personal and professional development, empowering you to make a genuine impact on our evolving product. Work in the ... View more
