Machine Learning Evaluation Engineer

About Us
We are Pepoltek an IT company building a portfolio of SaaS products used across education HR and enterprise operations including BIDYATek a school management platform for institutions across Bangladesh our Intelligent HR Solution a cloud-based HRMS used by growing companies and an Enterprise ERP suite for supply chain and financial operations. Alongside our own products we work with software development partners across the globe to extend our engineering capacity on their behalf.

We are hiring a Machine Learning Evaluation Engineer for one of these partners a research-driven AI startup in New York building the memory and identity layer for agentic systems to own evaluation of their core product end to end.

The Role
Youll own evaluation of a complex multi-agent system end to end the instrument that tells a research-driven team whether their identity representations are actually getting better. The product works and it has real black-box behavior so the job is defining what better means for representations that change over time without ground truth and building the machinery that measures it. If you love constructing measurements from nothing reading traces to find whats actually broken and shipping the fix yourself this role is built for you.

What Youll Do

• Design the evals defining the scores and what better means for an entity representation that changes over time and keeping that definition current as the product and methods move
• Build the pipelines and harnesses including data ingestion labels versions reruns and judges so the team can ask a new question this week and get an answer this week
• Run the evals and harvest insights reading traces and results to find whats actually broken and proposing fixes that produce higher-fidelity representations
• Build simulation agents at scale raising the ceiling on how many entities can be modeled and measured at once
• Own the loop end to end from the question through the pipeline the rerun and the writeup without handing work off or waiting on someone elses sprint

What Youll Bring

• 3 years designing and owning LLM evaluation systems
• Experience designing building and running evals end to end for LLMs
• Production-quality Python including tests maintenance and release cadence
• Hands-on experience with PyTorch Hugging Face and Weights and Biases

Nice to Have

• Work on memory identity personalization agent evaluation or agentic optimization
• A shipped research record as first or co-author at NeurIPS ICML ICLR or equivalent or self-published open-source work of comparable quality
• Data pipeline tooling experience such as Hydra Kafka or SQL
• A degree in computer science mathematics statistics or a related quantitative field

Growth
You will own evaluation for the product end to end shaping what better means for the companys core measurement problem with direct influence over the research direction and methods the team relies on.

Compensation and Perks

• $220000 to $300000 base salary
• Competitive equity

Work Authorization
Candidates must have current work authorization in the United States without need for employer sponsorship.

Job Details

• Employment type: Full time
• Location: New York NY
• Workplace: On-site 5 days in-office
• Vacancies: 1

How to Apply
Apply with your updated Resume and a Cover letter or Motivation letter. We only move forward with candidates who clearly show us what they bring to the table.