Generalist Labeling

1. Role Overview

Mercor is collaborating with a leading AI lab on a short-term project focused on improving preference ranking models for conversational AI systems. Were seeking detail-oriented generalistsideally with prior experience in data labeling or content evaluationto assess and rank model outputs across a variety of domains. This opportunity is well-suited for professionals comfortable with nuanced judgment tasks and working independently in a remote setup.

2. Key Responsibilities

• Evaluate and compare AI-generated responses based on quality coherence and helpfulness

• Assign preference rankings to pairs or sets of model outputs

• Follow detailed labeling guidelines and adjust based on evolving criteria

• Provide brief written explanations for ranking decisions when required

• Flag edge cases or inconsistencies in task design or model output

3. Ideal Qualifications

• Prior experience in data labeling content moderation or preference ranking tasks

• Excellent critical thinking and reading comprehension skills

• Comfort working with evolving guidelines and ambiguity

• Strong attention to detail and consistency across repetitive tasks

• Availability for regular part-time work on a weekly basis

4. More About the Opportunity

• Remote and asynchronous set your own hours

• Expected commitment: 1020 hours/week

• Flexible workload depending on your availability and performance

5. Compensation & Contract Terms

• $2535/hour depending on experience and location

• Payments issued weekly via Stripe Connect

• This is a freelance engagement; youll be classified as an independent contractor

6. Application Process

• Submit your resume to get started

• Complete a short form to highlight your relevant experience

• You may be asked to complete a brief assessment to evaluate task fit

• Expect a response within 35 business days

7. About Mercor

• Mercor is a talent marketplace that connects top experts with leading AI labs and research organizations

• Our investors include Benchmark General Catalyst Adam DAngelo Larry Summers and Jack Dorsey

• Thousands of professionals across law engineering research and creative fields collaborate on contract-based projects shaping the future of AI