Hybrid

We are sharing a specialised full-time opportunity for experienced counsel-level attorneys with substantial legal practice experience and the ability to evaluate structure and improve advanced AI systems working on complex professional legal tasks.

This role works closely with AI research and programme teams to improve how advanced models reason about substantive legal work. Selected attorneys will review legal tasks and model outputs develop instruction specifications and reference solutions design challenging benchmarks and translate experienced legal judgement into clear reproducible evaluation standards.

Key Responsibilities

Legal Quality Review & Analysis

• Review complex legal knowledge-work tasks and AI-generated outputs for substantive accuracy and professional quality
• Identify weak reasoning overlooked issues unsupported conclusions and analyses that would not withstand professional legal scrutiny
• Evaluate legal work for completeness precision practical relevance and appropriate application of legal principles
• Assess model performance across sophisticated legal scenarios
• Provide precise written feedback explaining errors omissions and required improvements

Instruction Design & Reference Solutions

• Develop detailed instruction specifications for advanced legal AI tasks
• Produce high-quality reference solutions that establish clear standards for correct legal reasoning
• Design tasks reflecting authentic workflows encountered in professional legal practice
• Translate practitioner judgement into explicit teachable and reproducible evaluation criteria
• Ensure reference materials demonstrate appropriate substantive depth and analytical rigour

Legal Benchmark Development

• Design challenging legal benchmarks and evaluation sets
• Create tasks testing issue spotting substantive knowledge analytical reasoning and professional judgement
• Develop criteria that distinguish rigorous legal analysis from plausible but incomplete responses
• Refine evaluation materials based on model performance and research requirements
• Help develop legal-specific AI capabilities evaluation methodologies and supporting tools

Substantive Legal Expertise

• Apply specialist knowledge from one or more practice areas such as corporate and transactional law litigation and dispute resolution regulatory and compliance intellectual property employment and labour law or tax
• Evaluate complex matters using professional methodologies and applicable legal frameworks
• Identify nuanced legal issues that may be overlooked by generalist reviewers
• Apply judgement developed through direct ownership of significant legal matters
• Maintain standards consistent with experienced counsel-level practice

AI Research Collaboration & Calibration

• Collaborate with AI researchers programme teams and specialists across adjacent professional domains
• Participate in calibration processes to maintain consistent evaluation standards
• Apply hands-on experience with large language models to professional legal workflows
• Distinguish genuinely well-reasoned responses from outputs that are persuasive but substantively incorrect
• Communicate complex legal concepts clearly to technical and non-legal stakeholders

Ideal Profile

• Juris Doctor (JD) from an accredited law school
• Approximately 815 years of substantive post-qualification legal practice
• Professional experience within an established law firm corporate legal department regulatory body court or comparable legal institution
• Current or previous seniority at Counsel Senior Associate Senior In-House Counsel Assistant General Counsel or a comparable level
• Genuine specialisation in at least one substantive legal practice area
• Relevant expertise may include corporate and transactional law litigation and dispute resolution regulatory and compliance intellectual property employment and labour or tax
• Active admission to at least one U.S. state bar and good standing
• Direct ownership of significant matters transactions disputes regulatory work or advisory responsibilities
• Hands-on professional experience using large language models or AI tools
• Strong ability to distinguish rigorous legal reasoning from plausible but technically incorrect analysis
• Excellent written communication and ability to provide precise structured feedback
• Must live in the San Francisco Bay Area or be willing to relocate there before the engagement begins
• Must be able to work onsite with the assigned team multiple days per week when required

Engagement Details

• Full-time W-2 employment position
• Initial engagement of approximately 6 months
• 40 hours per week
• Hybrid working arrangement based in the San Francisco Bay Area
• Regular onsite participation is required when requested
• Candidates outside the Bay Area must be willing to relocate at their own expense before starting
• Relocation assistance is not provided
• Compensation: $75$110/hour
• Client-issued accounts and equipment will be provided as required
• Work will be conducted within assigned internal tools and workflows
• Responsibilities and project scope may evolve based on research and programme requirements
• Work must be completed without using confidential or proprietary information belonging to any current or former employer client institution or other third party

About the Platform

This opportunity is available through 24-MAG LLC. We connect experienced professionals with remote consulting opportunities across technical evaluation and project-based workstreams.

By submitting this application you acknowledge that your information may be processed by 24-MAG LLC for recruitment and opportunity matching in accordance with our Privacy Policy: