Strategic Projects Lead, Enterprise Evaluations

About the Role

Scales mission is to develop reliable AI systems for the worlds most important decisions.

Within the Enterprise BU we build production-grade GenAI applications for the worlds largest companies. For these organizations the stakes are high: if an application isnt useful accurate and safe it cannot go into production.

As a Strategic Projects Lead (SPL) for Enterprise Evaluations you will oversee the evaluations that determine if an application is ready for the real world. You will define what good looks like for complex GenAI apps curate the data needed to measure performance and serve as one of the final gatekeepers for production readiness.

This is a high-impact role for a technically curious operator who is equally comfortable debating a complex evaluation rubric with an engineer and communicating strategy to Fortune 500 customers. You must be obsessed with the gold standard for AI performance from the high-level approach to the granular details of data quality.

You Will:

• Partner with enterprise stakeholders and Scale project teams (Applied AI MLE Product) to translate business goals into concrete evaluation strategies.
• Co-design the frameworks rubrics and golden datasets determining the right mix of human expertise vs. automated evaluation designing evaluation strategies that capture the high-signal expert feedback that automated tools often miss.
• Determine the what how and why of human-in-the-loop data designing rubrics and scoring frameworks that allow human experts to provide high-signal feedback that automated tools often miss.
• Own operational scoping & execution converting complex evaluation needs into an executable plan (i.e. everything from staffing and contributor capacity planning to cost/pricing assumptions and final delivery)
• Orchestrate the end-to-end evaluation engine; while you wont personally label data you will own the execution of data labeling operations including ingest pipelines and version control.
• Identify and resolve operational challenges and technical blockers proactively unblocking yourself and the team by anticipating risks before they impact delivery.
• Analyze evaluation results to provide the final data-driven recommendation on whether an application is ready for production.
• Run open source LLM benchmarks and present insights and recommendations on model performance to engineering teams.
• Act as a cross-pollinator for the Enterprise BU identifying successful frameworks and turning them into repeatable SOPs that can be applied to other areas.

Ideally Youd Have:

• Strong technical background (ideal to have a degree in computer science and Python knowledge). At a minimum the role requires the ability to do data analytics using SQL or Python. You should be comfortable leveraging tools to automate tasks generate synthetic data or analyze evaluation results.
• 5 years of professional experience in a high-stakes operational role at a fast-growing tech company management consulting or investment banking.
• Strong problem solving capabilities (experience working on operational challenges or as a consultant is a plus).
• Systemic Thinking: You prefer building the machine over running manual workarounds. You are excited to build the operational infrastructure required to scale a new function and create methodologies that can be used and evolved over time.
• Research-Adjacent Interest: You enjoy keeping up with the fast-evolving GenAI landscape. You want to understand where human judgment is the gold standard and how to capture it using that knowledge to improve our evaluation strategies.
• Full-Stack Ownership:You have a proven track record of taking projects from 0 to 1. You have an entrepreneurial mindset and are excited about building things from scratch. You can handle high-level strategy but arent afraid to get deep into the data to ensure accuracy and reliability.

PLEASE NOTE:Our policy requires a 90-day waiting period before reconsidering candidates for the same role. This allows us to ensure a fair and thorough evaluation of all applicants.

About Us:

At Scale our mission is to develop reliable AI systems for the worlds most important decisions. Our products provide the high-quality data and full-stack technologies that power the worlds leading models and help enterprises and governments build deploy and oversee AI applications that deliver real impact. We work closely with industry leaders like Meta Cisco DLA Piper Mayo Clinic Time Inc. the Government of Qatar and U.S. government agencies including the Army and Air Force. We are expanding our team to accelerate the development of AI applications.

We believe that everyone should be able to bring their whole selves to work which is why we are proud to be an inclusive and equal opportunity workplace. We are committed to equal employment opportunity regardless of race color ancestry religion sex national origin sexual orientation age citizenship marital status disability status gender identity or Veteran status.

We are committed to working with and providing reasonable accommodations to applicants with physical and mental disabilities. If you need assistance and/or a reasonable accommodation in the application or recruiting process due to a disability please contact us at Please see the United States Department of Labors Know Your Rights poster for additional information.

We comply with the United States Department of Labors Pay Transparency provision.

PLEASE NOTE: We collect retain and use personal data for our professional business purposes including notifying you of job opportunities that may be of interest and sharing with our affiliates. We limit the personal data we collect to that which we believe is appropriate and necessary to manage applicants needs provide our services and comply with applicable laws. Any information we collect in connection with your application will be treated in accordance with our internal policies and programs designed to protect personal data. Please see our for additional information.