Research, Evals

Exa is building a search engine from scratch to serve every AI agent. We build massive-scale infrastructure to crawl the web train state-of-the-art embedding models to process it and design super high performant vector databases in rust to search over it. If you like compute we also own a $5M H200 GPU cluster (and soon 5xing that) and regularly spin up batchjobs with tens of thousands of machines.

We recently raised an $85M Series B from Benchmark and we are rapidly building the most intelligent search engine in history. Were high agency low-ego and united by the feeling that this is one of the last problems worth getting right.

The ML organization sits at the heart of this mission. We train foundational models for search. Our goal is to build systems that can instantly filter the worlds knowledge to exactly what you want no matter how complex your query. Basically put the web into an extremely powerful database.

And to do that well we need to measure what good search actually means. Thats where you come in.

Were looking for an ML evals engineer to design and build our eval stack at Exa. The role involves investigating how to evaluate search engines in an LLM world and then building the most comprehensive creative and effective eval suite. You will be deciding the future of search through the evals we choose to optimize for - your work will directly influence what the research team works on and shape the direction of the company.

Who You Are

• Have hands-on ML experience (training finetuning or evaluating models (bonus if related to embeddings or LLMs)

• Have strong engineering fundamentals and can build reliable systems (Python Rust distributed pipelines GPU/cluster jobs etc.)

• Enjoy diving into data via building eval sets inspecting edge cases designing creative measurement strategies

What You Could Do

• Write a manifesto of what perfect search means

• Design and implement evaluation frameworks that probe the limits of search

• Build scalable reliable eval pipelines that track regressions drift and quality signals across billions of documents

• Create golden datasets synthetic benchmarks agentic tasks and real-world test suites that reflect how developers agents and humans actually use Exa

• Partner closely with ML researchers data engineers infra engineers and product to shape the feedback loops that improve our search models

This is an in-person opportunity in San Francisco. Were happy to sponsor international candidates (e.g. STEM OPT OPT H1B O1 E3). In addition to premium healthcare benefits (medical dental vision) we also offer fertility benefits and a monthly wellness stipend to all of our employees.