Research Scientist - Science of Evaluations
Research Scientist - Science of Evaluations
Employer Active
Job Alert
You will be updated with latest job alerts via email
Share
Job Alert
You will be updated with latest job alerts via email
Job Description
About the AI Security Institute
The AI Security Institute is the largest team in a government dedicated to understanding AI capabilities and risks in the world.
Our mission is to equip governments with an empirical understanding of the safety of advanced AI systems. We conduct research to understand the capabilities and impacts of advanced AI and develop and test risk mitigations. We focus on risks with security implications including the potential of AI to assist with the development of chemical and biological weapons how it can be used to carry out cyberattacks enable crimes such as fraud and the possibility of loss of control.
The risks from AI are not scifi they are urgent. By combining the agility of a tech startup with the expertise and missiondriven focus of government were building a unique and innovative organisation to prevent AIs harms from impeding its potential.
The Science of Evaluations Team
AISIs Science of Evaluations team will conduct applied and foundational research focused on two areas at the core of our mission: (i) measuring existing frontier AI system capabilities and (ii) predicting the capabilities of a system before running an evaluation.
Measurement of Capabilities: the goal is to develop and apply rigorous scientific techniques for the measurement of frontier AI system capabilities so they are accurate robust and useful in decision making. This is a nascent area of research which supports one of AISIs core products: conducting tests of frontier AI systems and feeding back results insightsand recommendations to model developers and policy makers.
The team will be an independent voice on the quality of our testing reports and the limitations of our evaluations. You will collaborate closely with researchers and engineers from the workstreams who develop and run our evaluations getting into the details of their key strengths and weaknesses proposing improvements and developing techniques to get the most out of our results.
The key challenge is increasing the confidence in our claims about system capabilities based on solid evidence and analysis. Directions we are exploring include:
- Running internal red teaming of testing exercises and adversarial collaborations with the evaluations teams and developing sanity checks to ensure the claims made in our reports are as strong as possible. Example checks could include: performance as a function of context length auditing areas with surprising model performance checking for soft refusal performance issues and efficient comparisons of system performance between predeployment and postdeployment testing.
- Running Indepth analyses of evaluations results to understand successes and failures and using these insights to create best practices for testing exercises.
- Developing our approach to uncertainty quantification and significance testing increasing statistical power (given time and token constraints).
- Developing methods for inferring model capabilities across given domains from task or benchmark success rates and assigning confidence levels to claims about capabilities.
Predictive Evaluations: the goal is to develop approaches to estimate the capabilities of frontier AI systems on tasks or benchmarks before they are run. Ideally we would be able to do this at some point early in the training process of a new model using information about the architecture dataset or training compute. This research aims to provide us with advance warning of models reaching a particular level of capability where additional safety mitigations may need to be put in place. This work is complementary to both safety casesan AISI foundational research effortand AISIs general evaluations work.
This topic is currently an area of active research (e.g. Ruan et al. 2024) and we believe it is poised to develop rapidly. We are particularly interested in developing predictive evaluations for complex longhorizon agent tasks since we believe this will be the most important type of evaluation as AI capabilities advance. You will help develop this field of research both by direct technical work and via collaborations with external experts partner organizations and policy makers.
Across both focus areas there will be significant scope to contribute to the overall vision and strategy of the science of evaluations team as an early hire. Youll receive coaching from your manager and mentorship from the research directors at AISI (including Geoffrey Irving and Yarin Gal) and work closely with talented Policy / Strategy leads and Research Engineers and Research Scientists.
Responsibilities
This role offers the opportunity to progress deep technical work at the frontier of AI safety and governance. Your work will include:
- Running internal red teaming of testing exercises and adversarial collaborations with the evaluations teams and developing sanity checks to ensure the claims made in our reports are as strong as possible.
- Conducting indepth analysis of evaluations methodology and results diagnosing possible sources of uncertainty or bias to improve our confidence in estimates of AI system capabilities.
- Improving the statistical analysis of evaluations results (e.g. model selection hypothesis testing significance testing uncertainty quantification).
- Developing and implementing internal bestpractices and protocols for evaluations and testing exercises.
- Staying well informed of the details and strengths and weaknesses of evaluations across domains in AISI and the state of the art in frontier AI evaluations research more broadly.
- Conducting research on predictive evaluations using the latest techniques from the published literature on AISIs internal evaluations as well as conducting novel research to improve these techniques.
- Writing and editing scientific reports and other materials aimed at diverse audiences focusing on synthesising empirical results and recommendations to key decisionmakers ensuring high standards in clarity precision and style.
Person Specification
To set you up for success we are looking for some of the following skills experience and attitudes but we are flexible in shaping the role to your background and expertise.
- Experience working within a worldleading team in machine learning or a related field (e.g. multiple first author publications at toptier conferences).
- Strong trackrecord of academic excellence (e.g. PhD in a technical field and/or spotlight papers at toptier conferences).
- Comprehensive understanding of large language models (e.g. GPT4). This includes both a broad understanding of the literature handson experience with designing and running evaluations scaling laws finetuning scaffolding prompting.
- Broad experience in empirical research methodologies potentially in fields outside of machine learning and statistical analysis (Tshaped: some deep knowledge lots of shallow knowledge in e.g. experimental design A/B testing Bayesian inference model selection hypothesis testing significance testing).
- Deeply care about methodological and statistical rigor balanced with pragmatism and willingness to get into the weeds.
- Experience with data visualization and presentation.
- Proven track record of excellent scientific writing and communication with ability to understand and communicate complex technical concepts for nontechnical stakeholders and synthesize scientific results into compelling narratives.
- Motivated to conduct technical research with an emphasis on direct policy impact rather than exploring novel ideas.
- Have a sense of mission urgency and responsibility for successdemonstrating problemsolving abilities and preparedness to acquire any missing knowledge necessary to get the job done.
- Ability to work autonomously and in a selfdirected way with high agencythriving in a constantly changing environment and a steadily growing team.
- Bring your own voice and experience but also an eagerness to support your colleagues together with a willingness to do whatever is necessary for the teams success.
Salary & Benefits
We are hiring individuals at all ranges of seniority and experience within this research unit and this advert allows you to apply for any of the roles within this range. Your dedicated talent partner will work with you as you move through our assessment process to explain our internal benchmarking process. The full range of salaries are available below salaries comprise of a base salary technical allowance plusadditional benefitsas detailed on this page.
- Level 3 Total Packageinclusiveof a base salary 35720 plus additional technical talent allowance of between
- Level 4 Total Packageinclusiveof a base salary 42495 plus additional technical talent allowance of between
- Level 5 Total Packageinclusiveof a base salary 55805 plus additional technical talent allowance of between
- Level 6 Total Packageinclusiveof a base salary 68770 plus additional technical talent allowance of between
- Level 7 Total Package 145000inclusiveof a base salary 68770 plus additional technical talent allowance of 76230
This role sits outside of the DDaT pay framework given the scope of this role requires in depth technical expertise in frontier AI safety robustness and advanced AI architectures.
There are a range of pension options available which can be found through the Civil Service website.
The Department for Science Innovation and Technology offers a competitive mix of benefits including:
- A culture of flexible working such as job sharing homeworking and compressed hours.
- A minimum of 25 days of paid annual leave increasing by 1 day per year up to a maximum of 30.
- An extensive range of learning & professional development opportunities which all staff are actively encouraged to pursue.
- Access to a range of retail travel and lifestyle employee discounts.
- The Department operates a discretionary hybrid working policy which provides for a combination of working hours from your place of work and from your home in the UK. The current expectation for staff is to attend the office or nonhome based location for 4060% of the time over the accounting period.
Selection Process
In accordance with the Civil Service Commission rules the following list contains all selection criteria for the interview process.
The interview process may vary candidate to candidate however you should expect a typical process to include some technical proficiency tests discussions with a crosssection of our team at AISI (including nontechnical staff) conversations with your workstream lead. The process will culminate in a conversation with members of the senior team here at AISI.
Candidates should expect to go through some or all of the following stages once an application has been submitted:
- Initial interview
- Technical take home test
- Second interview and review of take home test
- Third interview
- Final interview with members of the senior team
Required Experience
We select based on skills and experience regarding the following areas:
- Empirical research and statistical analysis
- Frontier AI model architecture training evaluation knowledge
- AI safety research knowledge
- Written communication
- Verbal communication
- Research problem selection
Additional Information
Internal Fraud Database
The Internal Fraud function of the Fraud Error Debt and Grants Function at the Cabinet Office processes details of civil servants who have been dismissed for committing internal fraud or who would have been dismissed had they not resigned. The Cabinet Office receives the details from participating government organisations of civil servants who have been dismissed or who would have been dismissed had they not resigned for internal fraud. In instances such as this civil servants are then banned for 5 years from further employment in the civil service. The Cabinet Office then processes this data and discloses a limited dataset back to DLUHC as a participating government organisations. DLUHC then carry out the pre employment checks so as to detect instances where known fraudsters are attempting to reapply for roles in the civil service. In this way the policy is ensured and the repetition of internal fraud is prevented. For more information please see Internal Fraud Register.
Security
Successful candidates must undergo a criminal record check and getbaseline personnel security standard (BPSS)clearancebefore they can be appointed. Additionally there is a strong preference for eligibility forcounterterrorist check (CTC)clearance. Some roles may require higher levels of clearance and we will state this by exception in the job advertisement.See our vetting charter here.Nationality requirements
We may be able to offer roles to applicant from any nationality or background. As such we encourage you to apply even if you do not meet the standard nationality requirements (opens in a new window).
Working for the Civil Service
Diversity and Inclusion
Employment Type
Full Time
