Research Evaluation Specialist Remote

Role Type: Contractor (Approximately 15 hours per week)
Location: Remote Worldwide

We are seeking experienced Research Evaluation Specialists with advanced academic qualifications or substantial research experience to contribute to the development of rigorous evaluation materials for advanced AI models.

In this role you will leverage your subject-matter expertise research skills and analytical reasoning to create challenging questions and evidence-based answers that assess AI model capabilities. You will conduct in-depth research verify findings using authoritative sources and refine evaluation materials to ensure accuracy complexity and clarity.

Prior experience in AI training data labeling or model evaluation is required along with strong research expertise in your respective discipline.

1. Develop original high-difficulty question-and-answer pairs that assess advanced AI model capabilities within your area of expertise.
2. Research verify and thoroughly document answers using primary sources scholarly literature and authoritative references including clear citations and supporting reasoning.
3. Design questions that require advanced reasoning methodological understanding and synthesis across multiple sources while avoiding oversimplified solutions.
4. Test questions against AI systems identify items that do not adequately challenge the models and iteratively increase complexity while maintaining factual accuracy.
5. Write questions and answers with exceptional precision and clarity eliminating ambiguity and ensuring that responses are logically defensible.
6. Incorporate reviewer feedback and consistently follow project guidelines and quality standards.
7. Maintain high standards of research integrity documentation and analytical rigor across all deliverables.

1. A completed Masters or PhD active doctoral candidacy or equivalent research experience as a specialist researcher assistant professor or professor in the relevant discipline.
2. A demonstrated record of scholarly research advanced subject-matter expertise and familiarity with primary academic literature.
3. Exceptional analytical thinking attention to detail and strong written communication skills in English.
4. Experience locating evaluating and cross-verifying information from authoritative sources.
5. Strong ability to work independently manage priorities and deliver high-quality research within a remote working environment.
6. Expertise in developing original challenging and methodologically sound questions that assess deep subject-matter understanding.
7. Required: Previous experience in AI training data labeling or AI model evaluation.

Compensation is output-based with payment provided for each task that meets the projects specifications and quality requirements. The time required to complete tasks may vary depending on individual expertise and workflow.

Minimum submission requirements apply and contributors are expected to complete a specified number of tasks each week.

• Hiring Timeline: Positions are typically filled within 48 hours.
• Availability: Candidates should be prepared to begin promptly.
• Project Start: Selected candidates are expected to start their first tasks within 2448 hours of completing onboarding.

The ideal candidate combines advanced academic or research expertise with practical experience in AI training or evaluation. They can develop sophisticated evaluation questions support answers with authoritative evidence identify weaknesses in AI-generated responses and deliver precise well-documented work independently while maintaining rigorous research and quality standards.