But these risks can be tackled.

With further progress in AI safety, we have an opportunity to develop AI for good: systems that are safe, ethical, and beneficial for everyone.

This article explains how you can help.

Thanks to Adam Gleave, Jacob Hilton and Rohin Shah for reviewing this article. And thanks to Charlie Rogers-Smith for his help, and his article on the topic — How to pursue a career in technical AI alignment.

Why AI safety technical research is high impact

As we’ve argued, in the next few decades, we might see the development of hugely powerful machine learning systems with the potential to transform society. This transformation could bring huge benefits — but only if we avoid the risks.

We think that the worst-case risks from AI systems arise in large part because AI systems could be misaligned — that is, they will aim to do things that we don’t want them to do. In particular, we think they could be misaligned in such a way that they develop (and execute) plans that pose risks to humanity’s ability to influence the world, even when we don’t want that influence to be lost.

We think this means that these future systems pose an existential threat to civilisation.

Even if we find a way to avoid this power-seeking behaviour, there are still substantial risks — such as misuse by governments or other actors — which could be existential threats in themselves.

There are many ways in which we could go about reducing the risks that these systems might pose. But one of the most promising may be researching technical solutions that prevent unwanted behaviour — including misaligned behaviour — from AI systems. (Finding a technical way to prevent misalignment in particular is known as the alignment problem.)

In the past few years, we’ve seen more organisations start to take these risks more seriously. Many of the leading companies developing AI — including Google DeepMind and OpenAI — have teams dedicated to finding these solutions, alongside academic research groups including at MIT, Cambridge, Carnegie Mellon University, and UC Berkeley.

That said, the field is still very new. We estimated in 2022 that there were only around 300 people working on technical approaches to reducing existential risks from AI systems,¹ making this a highly neglected field.

Finding technical ways to reduce this risk could be quite challenging. Any practically helpful solution must retain the usefulness of the systems (remaining economically competitive with less safe systems), and continue to work as systems improve over time (that is, it needs to be ‘scalable’). As we argued in our problem profile, it seems like it might be difficult to find viable solutions, particularly for modern ML (machine learning) systems.

(If you don’t know anything about ML, we’ve written a very very short introduction to ML, and we’ll go into more detail on how to learn about ML later in this article. Alternatively, if you do have ML experience, talk to our team — they can give you personalised career advice, make introductions to others working on these issues, and possibly even help you find jobs or funding opportunities.)

Although it seems hard, there are lots of avenues for more research — and the field really is very young, so there are new promising research directions cropping up all the time. So we think it’s moderately tractable, though we’re highly uncertain.

In fact, we’re uncertain about all of this and have written extensively about reasons we might be wrong about AI risk.

But, overall, we think that — if it’s a good fit for you — going into AI safety technical research may just be the highest-impact thing you can do with your career.

What does this path involve?

AI safety technical research generally involves working as a scientist or engineer at major AI companies, in academia, or in independent nonprofits.

These roles can be very hard to get. You’ll likely need to build up career capital before you end up in a high-impact role (more on this later, in the section on how to enter). That said, you may not need to spend a long time building this career capital — we’ve seen exceptionally talented people move into AI safety from other quantitative fields, sometimes in less than a year.

Most AI safety technical research falls on a spectrum between empirical research (experimenting with current systems as a way of learning more about what will work), and theoretical research (conceptual and mathematical research looking at ways of ensuring that future AI systems are safe).

No matter where on this spectrum you end up working, your career path might look a bit different depending on whether you want to aim at becoming a research lead — proposing projects, managing a team and setting direction — or a contributor — focusing on carrying out the research.

Finally, there are two slightly different roles you might aim for:

• In academia, research is often led by professors — the key distinguishing feature of being a professor is that you’ll also teach classes and mentor grad students (and you’ll definitely need a PhD).
• Many (but not all) contributor roles in empirical research are also engineers, often software engineers. Here, we’re focusing on software roles that directly contribute to AI safety research (and which often require some ML background) — we’ve written about software engineering more generally in a separate career review.

We think that research lead roles are probably higher-impact in general. But overall, the impact you could have in any of these roles is likely primarily determined by your personal fit for the role — see the section on how to predict your fit in advance.

Next, we’ll take a look at what working in each path might involve. Later, we’ll go into how you might enter each path.

What does work in the empirical AI safety path involve?

Empirical AI safety tends to involve teams working directly with ML models to identify any risks and develop ways in which they might be mitigated.

That means the work is focused on current ML techniques and techniques that might be applied in the very near future.

Practically, working on empirical AI safety involves lots of programming and ML engineering. You might, for example, come up with ways you could test the safety of existing systems, and then carry out these empirical tests.

You can find roles in empirical AI safety in industry and academia, as well as some in AI safety-focused nonprofits.

Particularly in academia, lots of relevant work isn’t explicitly labelled as being focused on existential risk — but it can still be highly valuable. For example, work in interpretability, adversarial examples, diagnostics and backdoor learning, among other areas, could be highly relevant to reducing the chance of an AI-related catastrophe.

We’re also excited by experimental work to develop safety standards that AI companies might adhere to in the future — for example, the work being carried out by METR.

To learn more about the sorts of research taking place at companies and labs focused on empirical AI safety, take a look at:

• OpenAI’s approach to alignment research
• Anthropic’s views on AI safety
• Redwood Research’s recent research highlights
• Publications from Google DeepMind’s safety team

While programming is central to all empirical work, generally, research lead roles will be less focused on programming; instead, they need stronger research taste and theoretical understanding. In comparison, research contributors need to be very good at programming and software engineering.

What does work in the theoretical AI safety path involve?

Theoretical AI safety is much more heavily conceptual and mathematical. Often it involves careful reasoning about the hypothetical behaviour of future systems.

Generally, the aim is to come up with properties that it would be useful for safe ML algorithms to have. Once you have some useful properties, you can try to develop algorithms with these properties (bearing in mind that to be practically useful these algorithms will have to end up being adopted by industry). Alternatively, you could develop ways of checking whether systems have these properties. These checks could, for example, help hold future AI products to high safety standards.

Many people working in theoretical AI safety will spend much of their time proving theorems or developing new mathematical frameworks. More conceptual approaches also exist, although they still tend to make heavy use of formal frameworks.

Some examples of research in theoretical AI safety include:

• Risks from learned optimisation in advanced machine learning systems by Hubinger et al.
• Eliciting latent knowledge by Christiano, Cotra and Xu.
• Formalizing the presumption of independence by Christiano, Neyman, and Xu
• Discovering agents by Kenton et al.
• Active reward learning from multiple teachers by Barnett et al.

There are generally fewer roles available in theoretical AI safety work, especially as research contributors. Theoretical research contributor roles exist at nonprofits (primarily the Alignment Research Center), as well as at some labs (for example, Anthropic’s work on conditioning predictive models and the Causal Incentives Working Group at Google DeepMind). Most contributor roles in theoretical AI safety probably exist in academia (for example, PhD students in teams working on projects relevant to theoretical AI safety).

Some exciting approaches to AI safety

There are lots of technical approaches to AI safety currently being pursued. Here are just a few of them:

• Scalably learning from human feedback. Examples include iterated amplification, AI safety via debate, building AI assistants that are uncertain about our goals and learn them by interacting with us, and other ways to get AI systems to report truthfully what they know.
• Threat modelling. An example of this work would be demonstrating the possibility of (allowing us to study) dangerous capabilities, like deceptive or manipulative AI systems. This approach splits into work that evaluates whether a model has dangerous capabilities (like the work of METR in evaluating GPT-4), and work that evaluates whether a model would cause harm in practice (like Anthropic’s research into the behaviour of large language models and this paper on goal misgeneralisation). It can also include work to find ‘model organisms of misalignment’, in the hope of better understanding their dangers.
• Work to figure out how to control powerful AI systems – preventing them from causing harm even if they are unsafe. Read more in this blogpost from the team at Redwood Research working on control.
• Interpretability research. This work involves studying why AI systems do what they do and trying to put it into human-understandable terms. For example, this paper examined how AlphaZero learns chess, and this paper looked into finding latent knowledge in language models without supervision. This category also includes mechanistic interpretability — for example, Zoom In: An Introduction to Circuits by Olah et al.. For more, see this survey paper, as well as Hubinger’s a transparency and interpretability tech tree, and Nanda’s A Longlist of Theories of Impact for Interpretability for overviews of of how interpretability research could reduce existential risk from AI.
• Other anti-misuse research to reduce the risks of catastrophe caused by misuse of systems. For example, this work includes training AIs so they’re hard to use for dangerous purposes. (Note there’s lots of overlap with the other work on this list).
• Research to increase the robustness of neural networks. This work involves ensuring that the sorts of behaviour neural networks display when exposed to one set of inputs continues when exposed to inputs they haven’t previously been exposed to, in order to prevent AI systems changing to unsafe behaviour. See section 2 of Unsolved Problems in AI safety for more.
• Work to build cooperative AI. Find ways to ensure that even if individual AI systems seem safe, they don’t produce bad outcomes through interacting with other sociotechnical systems. For more, see Open Problems in Cooperative AI by Dafoe et al. or the Cooperative AI Foundation. This seems particularly relevant for the reduction of ‘s-risks.’
• More generally, there are some unified safety plans. For more, see Hubinger’s 11 possible proposals for building safe advanced AI, or Karnofsky’s How might we align transformative AI if it’s developed very soon.²

It’s worth noting that there are many approaches to AI safety, and people in the field strongly disagree on what will or won’t work.

This means that, once you’re working in the field, it can be worth being charitable and careful not to assume that others’ work is unhelpful just because it seemed so on a quick skim. You should probably be uncertain about your own research agenda as well.

What’s more, as we mentioned earlier, lots of relevant work across all these areas isn’t explicitly labelled ‘safety.’

So it’s important to think carefully about how or whether any particular research helps reduce the risks that AI systems might pose.

What are the downsides of this career path?

AI safety technical research is not the only way to make progress on reducing the risks that future AI systems might pose. Also, there are many other pressing problems in the world that aren’t the possibility of an AI-related catastrophe, and lots of careers that can help with them. If you’d be a better fit working on something else, you should probably do that.

Beyond personal fit, there are a few other downsides to the career path:

• It can be very competitive to enter (although once you’re in, the jobs are well paid, and there are lots of backup options).
• You need quantitative skills — and probably programming skills.
• The work is geographically concentrated in just a few places (mainly the California Bay Area and London, but there are also opportunities in places with top universities such as Oxford, New York, Pittsburgh, and Boston). That said, remote work is increasingly possible at many research labs.
• It might not be very tractable to find good technical ways of reducing the risk. Although assessments of its difficulty vary, and while making progress is almost certainly possible, it may be quite hard to do so. This reduces the impact that you could have working in the field. That said, if you start out in technical work you might be able to transition to governance work, since that often benefits from technical training and experience with the industry, which most people do not have.)
• Relatedly, there’s lots of disagreement in the field about what could work; you’ll probably be able to find at least some people who think what you’re working on is useless, whatever you end up doing.
• Most importantly, there’s some risk of doing harm. While gaining career capital, and while working on the research itself, you’ll have to make difficult decisions and judgement calls about whether you’re working on something beneficial (see our anonymous advice about working in roles that advance AI capabilities). There’s huge disagreement on which technical approaches to AI safety might work — and sometimes this disagreement takes the form of thinking that a strategy will actively increase existential risks from AI.

Finally, we’ve written more about the best arguments against AI being pressing in our problem profile on preventing an AI-related catastrophe. If those are right, maybe you could have more impact working on a different issue.

How much do AI safety technical researchers earn?

Many technical researchers work at companies or small startups that pay wages competitive with the Bay Area and Silicon Valley tech industry, and even smaller organisations and nonprofits will pay competitive wages to attract top talent. The median compensation for a software engineer in the San Francisco Bay area was $222,000 per year in 2020.³ (Read more about software engineering salaries).

This $222,000 median may be an underestimate, as AI roles, especially in top AI companies that are rapidly scaling up their work in AI, often pay better than other tech jobs, and the same applies to safety researchers — even those in nonprofits.

However, academia has lower salaries than industry in general, and we’d guess that AI safety research roles in academia pay less than commercial labs and nonprofits.

Examples of people pursuing this path

How to predict your fit in advance

You’ll generally need a quantitative background (although not necessarily a background in computer science or machine learning) to enter this career path.

There are two main approaches you can take to predict your fit, and it’s helpful to do both:

• Try it out: try out the first few steps in the section below on learning the basics. If you haven’t yet, try learning some python, as well as taking courses in linear algebra, calculus, and probability. And if you’ve done that, try learning a bit about deep learning and AI safety. Finally, the best way to try this out for many people would be to actually get a job as a (non-safety) ML engineer (see more in the section on how to enter).
• Talk to people about whether it would be a good fit for you: If you want to become a technical researcher, our team probably wants to talk to you. We can give you 1-1 advice, for free. If you know anyone working in the area (or something similar), discuss this career path with them and ask for their honest opinion. You may be able to meet people through our community. Our advisors can also help make connections.

It can take some time to build expertise, and enjoyment can follow expertise — so be prepared to take some time to learn and practice before you decide to switch to something else entirely.

If you’re not sure what roles you might aim for longer term, here are a few rough ways you could make a guess about what to aim for, and whether you might be a good fit for various roles on this path:

• Testing your fit as an empirical research contributor: In a blog post about hiring for safety researchers, the Google DeepMind team said “as a rough test for the Research Engineer role, if you can reproduce a typical ML paper in a few hundred hours and your interests align with ours, we’re probably interested in interviewing you.”
  • Looking specifically at software engineering, one hiring manager at Anthropic said that if you could, with a few weeks’ work, write a complex new feature or fix a very serious bug in a major ML library, they’d want to interview you straight away. (Read more.)
• Testing your fit for theoretical research: If you could have got into a top 10 maths or theoretical computer science PhD programme if you’d optimised your undergrad to do so, that’s a decent indication of your fit (and many researchers in fact have these PhDs). The Alignment Research Center (one of the few organisations that hires for theoretical research contributors, as of 2023) said that they were open to hiring people without any research background. They gave four tests of fit: creativity (e.g. you may have ideas for solving open problems in the field, like Eliciting Latent Knowledge); experience designing algorithms, proving theorems, or formalising concepts; broad knowledge of maths and computer science; and having thought a lot about the AI alignment problem in particular.
• Testing your fit as a research lead (or for a PhD): The vast majority of research leads have a PhD. Also, many (but definitely not all) AI safety technical research roles will require a PhD — and if they don’t, having a PhD (or being the sort of person that could get one) would definitely help show that you’re a good fit for the work. To get into a top 20 machine learning PhD programme, you’d probably need to publish something like a first author workshop paper, as well as a third author conference paper at a major ML conference (like NeurIPS or ICML). (Read more about whether you should do a PhD).

Read our article on personal fit to learn more about how to assess your fit for the career paths you want to pursue.

How to enter

You might be able to apply for roles right away — especially if you meet, or are near meeting, the tests we just looked at — but it also might take you some time, possibly several years, to skill up first.

So, in this section, we’ll give you a guide to entering technical AI safety research. We’ll go through four key questions:

1. How to learn the basics
2. Whether you should do a PhD
3. How to get a job in empirical research
4. How to get a job in theoretical research

Hopefully, by the end of the section, you’ll have everything you need to get going.

Learning the basics

To get anywhere in the world of AI safety technical research, you’ll likely need a background knowledge of coding, maths, and deep learning.

You might also want to practice enough to become a decent ML engineer (although this is generally more useful for empirical research), and learn a bit about safety techniques in particular (although this is generally more useful for empirical research leads and theoretical researchers).

We’ll go through each of these in turn.

Learning to program

You’ll probably want to learn to code in python, because it’s the most widely used language in ML engineering.

The first step is probably just trying it out. As a complete beginner, you can write a Python program in less than 20 minutes that reminds you to take a break every two hours. Don’t be discouraged if your code doesn’t work the first time — that’s what normally happens when people code!

Once you’ve done that, you have a few options:

• Teach yourself to program. Try working through a free beginner course like Automate the boring stuff with Python by Al Seigart. There also are many great introductory computer science and programming courses online, including: Udacity’s Intro to Computer Science, MIT’s Introduction to Computer Science and Programming, and Stanford’s Programming Methodology. Then, try finding something you want to build, and building it — or getting involved in an open-source project. For interview practice, try leetcode or TopCoder, or the exercises in Cracking the Coding Interview by Gayle McDowell.
• Take a college course. If you’re in university, this is a great option because it allows you to learn programming while the opportunity cost of your time is lower. You can even consider majoring in computer science (or another subject involving lots of programming).
• Learn on the job. If you can find internships, you’ll gain practical experience and skills you otherwise wouldn’t pick up from academic degrees.
• Go to a bootcamp. Coding bootcamps are focused on taking people with little knowledge of programming to as highly paid a job as possible within a couple of months — though some claim the long-term prospects are not as good because you lack a deep understanding of computer science. Course Report is a great guide to choosing a bootcamp. Be careful to avoid low-quality bootcamps. You can also find online bootcamps — for people completely new to programming — focused on ML, like Udemy’s Python for Data Science and Machine Learning Bootcamp.

You can read more about learning to program — and how to get your first job in software engineering (if that’s the route you want to take) — in our career review on software engineering.

Learning the maths

The maths of deep learning relies heavily on calculus and linear algebra, and statistics can be useful too — although generally learning the maths is much less important than programming and basic, practical ML.

We’d generally recommend studying a quantitative degree (like maths, computer science or engineering), most of which will cover all three areas pretty well.

If you want to actually get good at maths, you have to be solving problems. So, generally, the most useful thing that textbooks and online courses provide isn’t their explanations — it’s a set of exercises to try to solve, in order, with some help if you get stuck.

If you want to self-study (especially if you don’t have a quantitative degree) here are some possible resources:

• Calculus: 3blue1brown’s video series on calculus could be a good place to start. You may also be able to follow recorded university courses: MIT’s single variable calculus (which requires only high school algebra and trigonometry) followed by MIT’s course in vector and multivariable calculus.
• Linear algebra: Again, we’d suggest 3blue1brown’s video series on linear algebra as a place to start. In his post about technical alignment careers, Rogers-Smith recommends Linear Algebra Done Right by Sheldon Axler. Finally, if you prefer lectures, try MIT’s undergraduate course in linear algebra (although note that this course assumes knowledge of multivariate calculus).
• Probability: Take a look at MIT’s undergraduate course in probability and random variables.

You might be able to find resources that cover all these areas, like Imperial College’s Mathematics for Machine Learning.

Learning basic machine learning

You’ll likely need to have a decent understanding of how AI systems are currently being developed. This will involve learning about machine learning and neural networks, before diving into any specific subfields of deep learning.

Again, there’s the option of covering this at university. If you’re currently at college, it’s worth checking if you can take an ML course even if you’re not majoring in computer science.

There’s one important caveat here: you’ll learn a huge amount on the job, and the amount you’ll need to know in advance for any role or course will vary hugely! Not even top academics know everything about their fields. It’s worth trying to find out how much you’ll need to know for the role you want to do before you invest hundreds of hours into learning about ML.

With that caveat in mind, here are some suggestions of places you might start if you want to self-study the basics:

• 3blue1brown’s series on neural networks is a really great place to start for beginners.
• When I was learning, I used Neural Networks and Deep Learning — it’s an online textbook, good if you’re familiar with the maths, with some helpful exercises as well.
• Online intro courses like fast.ai (focused on practical applications), Full Stack Deep Learning, and the various courses at deeplearning.ai.
• For more detail, see university courses like MIT’s *Introduction to Machine Learning, NYU’s Deep Learning for even more detail. We’d also recommend Google DeepMind’s lecture series.

PyTorch is a very common package used for implementing neural networks, and probably worth learning! When I was first learning about ML, my first neural network was a 3-layer convolutional neural network with L2 regularisation classifying characters from the MNIST database. This is a pretty common first challenge, and a good way to learn PyTorch.

Learning about AI safety

If you’re going to work as an AI safety researcher, it usually helps to know about AI safety.

This isn’t always true — some engineering roles won’t require much knowledge of AI safety. But even then, knowing the basics will probably help land you a position, and can also help with things like making difficult judgement calls and avoiding doing harm. And if you want to be able to identify and do useful work, you’ll need to learn about the field eventually.

Because the field is still so new, there probably aren’t (yet) university courses you can take. So you’ll need to do some self-study. Here are some places you might start:

• Section 3 of our problem profile about preventing an AI-related catastrophe provides an introduction to the problems that AI safety attempts to solve (with a particular focus on alignment).
• Rob Miles’ YouTube channel is full of popular and well-explained introductory videos that don’t need much background knowledge of ML.
• AXRP – the AI X-risk Research Podcast — is full of in-depth (and enjoyable) conversations with researchers about their research.
• The ARENA course and curriculum provide a strong foundation for empirical AI safety research and engineering.
• The courses from AGI Safety Fundamentals, in particular the AI Alignment Course, which provides an introduction to research on the alignment problem.
• Intro to ML Safety, a course from the Center for AI Safety focuses on withstanding hazards (“robustness”), identifying hazards (“monitoring”), and reducing systemic hazards (“systemic safety”), as well as alignment.

For more suggestions — especially when it comes to reading about the nature of the risks we might face from AI systems — take a look at the top resources to learn more from our problem profile.

Should you do a PhD?

Some technical research roles will require a PhD — but many won’t, and PhDs aren’t the best option for everyone.

The main benefit of doing a PhD is probably practising setting and carrying out your own research agenda. As a result, getting a PhD is practically the default if you want to be a research lead.

That said, you can also become a research lead without a PhD — in particular, by transitioning from a role as a research contributor. At some large labs, the boundary between being a contributor and a lead is increasingly blurry.

Many people find PhDs very difficult. They can be isolating and frustrating, and take a very long time (4–6 years). What’s more, both your quality of life and the amount you’ll learn will depend on your supervisor — and it can be really difficult to figure out in advance whether you’re making a good choice.

So, if you’re considering doing a PhD, here are some things to consider:

• Your long-term vision: If you’re aiming to be a research lead, that suggests you might want to do a PhD — the vast majority of research leads have PhDs. If you mainly want to be a contributor (e.g. an ML or software engineer), that suggests you might not. If you’re unsure, you should try doing something to test your fit for each, like trying a project or internship. You might try a pre-doctoral research assistant role — if the research you do is relevant to your future career, these can be good career capital, whether or not you do a PhD.
• The topic of your research: It’s easy to let yourself become tied down to a PhD topic you’re not confident in. If the PhD you’re considering would let you work on something that seems useful for AI safety, it’s probably — all else equal — better for your career, and the research itself might have a positive impact as well.
• Mentorship: What are the supervisors or managers like at the opportunities open to you? You might be able to find ML engineering or research roles in industry where you could learn much more than you would in a PhD — or vice versa. When picking a supervisor, try reaching out to the current or former students of a prospective supervisor and asking them some frank questions. (Also, see this article on how to choose a PhD supervisor.)
• Your fit for the work environment: Doing a PhD means working on your own with very little supervision or feedback for long periods of time. Some people thrive in these conditions! But some really don’t and find PhDs extremely difficult.

Read more in our more detailed (but less up-to-date) review of machine learning PhDs.

It’s worth remembering that most jobs don’t need a PhD. And for some jobs, especially empirical research contributor roles, even if a PhD would be helpful, there are often better ways of getting the career capital you’d need (for example, working as a software or ML engineer). We’ve interviewed two ML engineers who have had hugely successful careers without doing a PhD.

Whether you should do a PhD doesn’t depend (much) on timelines

We think it’s plausible that we will develop AI that could be hugely transformative for society by the end of the 2030s.

All else equal, that possibility could argue for trying to have an impact right away, rather than spending five (or more) years doing a PhD.

Ultimately, though, how well you, in particular, are suited to a particular PhD is probably a much more important factor than when AI will be developed.

That is to say, we think the increase in impact caused by choosing a path that’s a good fit for you is probably larger than any decrease in impact caused by delaying your work. This is in part because the spread in impact caused by the specific roles available to you, as well as your personal fit for them, is usually very large. Some roles (especially research lead roles) will just require having a PhD, and others (especially more engineering-heavy roles) won’t — and people’s fit for these paths varies quite a bit.

We’re also highly uncertain about estimates about when we might develop transformative AI. This uncertainty reduces the expected cost of any delay.

Most importantly, we think PhDs shouldn’t be thought of as a pure delay to your impact. You can do useful work in a PhD, and generally, the first couple of years in any career path will involve a lot of learning the basics and getting up to speed. So if you have a good mentor, work environment, and choice of topic, your PhD work could be as good as, or possibly better than, the work you’d do if you went to work elsewhere early in your career. And if you suddenly receive evidence that we have less time than you thought, it’s relatively easy to drop out.

There are lots of other considerations here — for a rough overview, and some discussion, see this post by 80,000 Hours advisor Alex Lawsen, as well as the comments.

Overall, we’d suggest that instead of worrying about a delay to your impact, think instead about which longer-term path you want to pursue, and how the specific opportunities in front of you will get you there.

How to get into a PhD

ML PhDs can be very competitive. To get in, you’ll probably need a few publications (as we said above, something like a first author workshop paper, as well as a third author conference paper at a major ML conference (like NeurIPS or ICML), and references, probably from ML academics. (Although publications also look good whatever path you end up going down!)

To end up at that stage, you’ll need a fair bit of luck, and you’ll also need to find ways to get some research experience.

One option is to do a master’s degree in ML, although make sure it’s a research masters — most ML master’s degrees primarily focus on preparation for industry.

Even better, try getting an internship in an ML research group. Opportunities include RISS at Carnegie Mellon University, UROP at Imperial College London, the Aalto Science Institute international summer research programme, the Data Science Summer Institute, the Toyota Technological Institute intern programme and MILA. You can also try doing an internship specifically in AI safety, for example at CHAI. However, there are sometimes disadvantages to doing internships specifically in AI safety directly — in general, it may be harder to publish and mentorship might be more limited.

Another way of getting research experience is by asking whether you can work with researchers. If you’re already at a top university, it can be easiest to reach out to people working at the university you’re studying at.

PhD students or post-docs can be more responsive than professors, but eventually, you’ll want a few professors you’ve worked with to provide references, so you’ll need to get in touch. Professors tend to get lots of cold emails, so try to get their attention! You can try:

• Getting an introduction, for example from a professor who’s taught you
• Mentioning things you’ve done (your grades, relevant courses you’ve taken, your GitHub, any ML research papers you’ve attempted to replicate as practice)
• Reading some of their papers and the main papers in the field, and mention them in the email
• Applying for funding that’s available to students who want to work in AI safety, and letting people know you’ve got funding to work with them

Ideally, you’ll find someone who supervises you well and has time to work with you (that doesn’t necessarily mean the most famous professor — although it helps a lot if they’re regularly publishing at top conferences). That way, they’ll get to know you, you can impress them, and they’ll provide an amazing reference when you apply for PhDs.

It’s very possible that, to get the publications and references you’ll need to get into a PhD, you’ll need to spend a year or two working as a research assistant, although these positions can also be quite competitive.

This guide by Adam Gleave also goes into more detail on how to get a PhD, including where to apply and tips on the application process itself. We discuss ML PhDs in more detail in our career review on ML PhDs (though it’s outdated compared to this career review).

Getting a job in empirical AI safety research

Ultimately, the best way of learning to do empirical research — especially in contributor and engineering-focused roles — is to work somewhere that does both high-quality engineering and cutting-edge research.

The top three companies are probably Google DeepMind (who offer internships to students), OpenAI (who have a 6-month residency programme) and Anthropic. (Working at a leading AI company carries with it some risk of doing harm, so it’s important to think carefully about your options. We’ve written a separate article going through the major relevant considerations.)

To end up working in an empirical research role, you’ll probably need to build some career capital.

Whether you want to be a research lead or a contributor, it’s going to help to become a really good software engineer. The best ways of doing this usually involve getting a job as a software engineer at a big tech company or at a promising startup. (We’ve written an entire article about becoming a software engineer.)

Many roles will require you to be a good ML engineer, which means going further than just the basics we looked at above. The best way to become a good ML engineer is to get a job doing ML engineering — and the best places for that are probably leading AI companies.

For roles as a research lead, you’ll need relatively more research experience. You’ll either want to become a research contributor first, or enter through academia (for example by doing a PhD).

All that said, it’s important to remember that you don’t need to know everything to start applying, as you’ll inevitably learn loads on the job — so do try to find out what you’ll need to learn to land the specific roles you’re considering.

How much experience do you need to get a job? It’s worth reiterating the tests we looked at above for contributor roles:

• In a blog post about hiring for safety researchers, the DeepMind team said “as a rough test for the Research Engineer role, if you can reproduce a typical ML paper in a few hundred hours and your interests align with ours, we’re probably interested in interviewing you.”
• Looking specifically at software engineering, one hiring manager at Anthropic said that if you could, with a few weeks’ work, write a new feature or fix a serious bug in a major ML library, they’d want to interview you straight away. (Read more.)

In the process of getting this experience, you might end up working in roles that advance AI capabilities. There are a variety of views on whether this might be harmful — so we’d suggest reading our article about working at leading AI companies and our article containing anonymous advice from experts about working in roles that advance capabilities. It’s also worth talking to our team about any specific opportunities you have.

If you’re doing another job, or a degree, or think you need to learn some more before trying to change careers, there are a few good ways of getting more experience doing ML engineering that go beyond the basics we’ve already covered:

• Getting some experience in software / ML engineering. For example, if you’re doing a degree, you might try an internship as a software engineer during the summer. DeepMind offer internships for students with at least two years of study in a technical subject,
• Replicating papers. One great way of getting experience doing ML engineering, is to replicate some papers in whatever sub-field you might want to work in. Richard Ngo, an AI governance researcher at OpenAI, has written some advice on replicating papers. But bear in mind that replicating papers can be quite hard — take a look at Amid Fish’s blog on what he learned replicating a deep RL paper. Finally, Rogers-Smith has some suggestions on papers to replicate. If you do spend some time replicating papers, remember that when you get to applying for roles, it will be really useful to be able to prove you’ve done the work. So try uploading your work to GitHub, or writing a blog on your progress. And if you’re thinking about spending a long time on this (say, over 100 hours), try to get some feedback on the papers you might replicate before you start — you could even reach out to a lab or company you want to work for.
• Taking or following a more in-depth course in empirical AI safety research. Redwood Research ran the MLAB bootcamp, and you can apply for access to their curriculum here. You could also take a look at this Deep Learning Curriculum by Jacob Hilton, a researcher at the Alignment Research Center — although it’s probably very challenging without mentorship.⁴ The Alignment Research Engineer Accelerator is a program that uses this curriculum. Mentors in the ML Alignment & Theory Scholars Program primarily focus on empirical research.
• Learning about a sub-field of deep learning. In particular, we’d suggest natural language processing (in particular transformers — see this lecture as a starting point) and reinforcement learning (take a look at Pong from Pixels by Andrej Karpathy, and OpenAI’s Spinning up in Deep RL). Try to get to the point where you know about the most important recent advances.

Finally, Athena is an AI alignment mentorship program for women with a technical background looking to get jobs in the alignment field.

Getting a job in theoretical AI safety research

There are fewer jobs available in theoretical AI safety research, so it’s harder to give concrete advice. Having a maths or theoretical computer science PhD isn’t always necessary, but is fairly common among researchers in industry, and is pretty much required to be an academic.

If you do a PhD, ideally it’d be in an area at least somewhat related to theoretical AI safety research. For example, it could be in probability theory as applied to AI, or in theoretical CS (look for researchers who publish in COLT or FOCS).

Alternatively, one path is to become an empirical research lead before moving into theoretical research.

Compared to empirical research, you’ll need to know relatively less about engineering, and relatively more about AI safety as a field.

Once you’ve done the basics, one possible next step you could try is reading papers from a particular researcher, or on a particular topic, and summarising what you’ve found.

You could also try spending some time (maybe 10–100 hours) reading about a topic and then some more time (maybe another 10–100 hours) trying to come up with some new ideas on that topic. For example, you could try coming up with proposals to solve the problem of eliciting latent knowledge. Alternatively, if you wanted to focus on the more mathematical side, you could try having a go at the assignment at the end of this lecture by Michael Cohen, a grad student at the University of Oxford.

If you want to enter academia, reading a ton of papers seems particularly important. Maybe try writing a survey paper on a certain topic in your spare time. It’s a great way to master a topic, spark new ideas, spot gaps, and come up with research ideas. When applying to grad school or jobs, your paper is a fantastic way to show you love research so much you do it for fun.

Other ways to get more concrete experience include doing research internships, working as a research assistant, or doing a PhD, all of which we’ve written about above, in the section on whether and how you can get into a PhD programme.

One note is that a lot of people we talk to try to learn independently. This can be a great idea for some people, but is fairly tough for many, because there’s substantially less structure and mentorship.

Key organisations

AI companies that have empirical technical safety teams, or are focused entirely on safety:

AI organisations that have empirical technical safety teams, or are focused entirely on safety:

• Anthropic is a safety-focused AI company working on building interpretable and safe AI systems. They focus on empirical AI safety research. Anthropic cofounders Daniela and Dario Amodei gave an interview about the lab on the Future of Life Institute podcast. On our podcast, we spoke to Chris Olah, who leads Anthropic’s research into interpretability, and Nova DasSarma, who works on systems infrastructure at Anthropic.
• METR works on assessing whether cutting-edge AI systems could pose catastrophic risks to civilization, including early-stage, experimental work to develop techniques, and evaluating systems produced by Anthropic and OpenAI.
• The UK government’s AI Safety Institute is conducting research to assess the risks posed by advanced AI systems. It also coordinates with various companies, governments, and other actors, and it works to inform policymakers and shape safety practices around AI development globally.
• The Center for AI Safety is a nonprofit that does technical research and promotion of safety in the wider machine learning community.
• FAR AI is a research nonprofit that incubates and accelerates research agendas that are too resource-intensive for academia but not yet ready for commercialisation by industry, including research in adversarial robustness, interpretability and preference learning.
• Apollo Research is a non-profit that aims to develop AI model evaluation processes to detect signs of misalignment and deception. It works on interpretability, behaviour tests, and fine-tuning, and it aims to provide technical support to lawmakers looking to govern advanced AI.
• Google DeepMind is probably the largest and most well-known research group developing general artificial machine intelligence, and is famous for its work creating AlphaGo, AlphaZero, and AlphaFold. It is not principally focused on safety, but has two teams focused on AI safety, with the Scalable Alignment Team focusing on aligning existing state-of-the-art systems, and the Alignment Team focused on research bets for aligning future systems.
• OpenAI, founded in 2015, is a company that is trying to build artificial general intelligence that is safe and benefits all of humanity. OpenAI is well known for its language models like GPT-4. Like DeepMind, it is not principally focused on safety, but has a preparedness team and a governance team.
• Ought is a machine learning lab building Elicit, an AI research assistant. Their aim is to align open-ended reasoning by learning human reasoning steps, and to direct AI progress towards helping with evaluating evidence and arguments.
• Redwood Research is an AI safety research organisation, whose first big project attempted to make sure language models (like GPT-3) produce output following certain rules with very high probability, in order to address failure modes too rare to show up in standard training.

Theoretical / conceptual AI safety labs:

• The Alignment Research Center (ARC) is attempting to produce alignment strategies that could be adopted in industry today while also being able to scale to future systems. They focus on conceptual work, developing strategies that could work for alignment and which may be promising directions for empirical work, rather than doing empirical AI work themselves. Their first project was releasing a report on Eliciting Latent Knowledge, the problem of getting advanced AI systems to honestly tell you what they believe (or ‘believe’) about the world. On our podcast, we interviewed ARC founder Paul Christiano about his research (before he founded ARC).
• The Center on Long-Term Risk works to address worst-case risks from advanced AI. They focus on conflict between AI systems.
• The Machine Intelligence Research Institute was one of the first groups to become concerned about the risks from machine intelligence in the early 2000s, and its team has published a number of papers on safety issues and how to resolve them.
• Some teams in commercial labs also do some more theoretical and conceptual work on alignment, such as Anthropic’s work on conditioning predictive models and the Causal Incentives Working Group at Google DeepMind.

AI safety in academia (a very non-comprehensive list; while the number of academics explicitly and publicly focused on AI safety is small, it’s possible to do relevant work at a much wider set of places):

• The Algorithmic Alignment Group in the Computer Science and Artificial Intelligence Laboratory at MIT, led by Dylan Hadfield-Menell
• The Center for Human-Compatible AI at UC Berkeley, led by Stuart Russell, focuses on academic research to ensure AI is safe and beneficial to humans. (Our podcast with Stuart Russell examines his approach to provably beneficial AI.)
• Jacob Steinhardt’s research group in the Department of Statistics at UC Berkeley
• The NYU Alignment research Group led by Sam Bowman
• The Krueger AI Safety Lab at the University of Cambridge led by David Krueger
• The Tegmark AI Safety Group led by Max Tegmark of MIT
• The Foundations of Cooperative AI Lab at Carnegie Mellon University
• The Alignment of Complex Systems research group at Charles University, Prague

There are also some AI safety offices that can host independent researchers. These include LISA in London and FAR Labs in Berkeley, California.

Find a job in this path

If you think you might be a good fit for this path and you’re ready to start looking at job opportunities that are currently accepting applications, see our curated list of opportunities for this path:

View all opportunities

Learn more about AI safety technical research

Top recommendations

• The AGI safety fundamentals technical alignment curriculum
• The 80,000 Hours Podcast on Artificial Intelligence (a collection of 10 key AI episodes from our podcast)
• Charlie Rogers-Smith’s step-by-step guide to AI safety careers (which this article is in large part based on) provides some helpful concrete advice, including ways you might get some funding to help you move into an AI safety technical research career.

Further recommendations

Other articles and resources

Here are some suggestions about where you could learn more:

• To help you get oriented in the field, we recommend the AI safety starter pack.
• Careers in Beneficial AI Research by Adam Gleave, CEO of FAR AI
• Our problem profile on AI risk
• This sequence of posts on AI safety technical alignment by Richard Ngo)
• Our career review of machine learning PhDs
• Our career review of software engineering
• Our career review of working at a leading AI lab

Read next:  Learn about other high-impact careers

Want to consider more paths? See our list of the highest-impact career paths according to our research.

Continue →

The post AI safety technical research appeared first on 80,000 Hours.

Note from the author: At its core, this problem profile tries to predict the future of technology. This is a notoriously difficult thing to do. In addition, there has been much less rigorous research into the risks from AI than into the other risks 80,000 Hours writes about (like pandemics or climate change).¹ That said, there is a growing field of research into the topic, which I’ve tried to reflect. For this article I’ve leaned especially on this report by Joseph Carlsmith at Open Philanthropy (also available as a narration), as it’s the most rigorous overview of the risk that I could find. I’ve also had the article reviewed by over 30 people with different expertise and opinions on the topic. (Almost all are concerned about advanced AI’s potential impact.)

Why do we think that reducing risks from AI is one of the most pressing issues of our time? In short, our reasons are:

1. Even before getting into the actual arguments, we can see some cause for concern — as many AI experts think there’s a small but non-negligible chance that AI will lead to outcomes as bad as human extinction.
2. We’re making advances in AI extremely quickly — which suggests that AI systems could have a significant influence on society, soon.
3. There are strong arguments that “power-seeking” AI could pose an existential threat to humanity² — which we’ll go through below.
4. Even if we find a way to avoid power-seeking, there are still other risks.
5. We think we can tackle these risks.
6. This work is neglected.

We’re going to cover each of these in turn, then consider some of the best counterarguments, explain concrete things you can do to help, and finally outline some of the best resources for learning more about this area.

If you’d like, you can watch our 10-minute video summarising the case for AI risk before reading further:

1. Many AI experts think there’s a non-negligible chance AI will lead to outcomes as bad as extinction

In May 2023, hundreds of prominent AI scientists — and other notable figures — signed a statement saying that mitigating the risk of extinction from AI should be a global priority.

So it’s pretty clear that at least some experts are concerned.

But how concerned are they? And is this just a fringe view?

We looked at four surveys of AI researchers who published at NeurIPS and ICML (two of the most prestigious machine learning conferences) from 2016, 2019, 2022 and 2023.³

It’s important to note that there could be considerable selection bias on surveys like this. For example, you might think researchers who go to the top AI conferences are more likely to be optimistic about AI, because they have been selected to think that AI research is doing good. Alternatively, you might think that researchers who are already concerned about AI are more likely to respond to a survey asking about these concerns.⁴

All that said, here’s what we found:

In all four surveys, the median researcher thought that the chances that AI would be “extremely good” was reasonably high: 20% in the 2016 survey, 20% in 2019, 10% in 2022, and 10% in 2023.⁵

Indeed, AI systems are already having substantial positive effects — for example, in medical care or academic research.

But in all four surveys, the median researcher also estimated small — and certainly not negligible — chances that AI would be “extremely bad (e.g. human extinction)”: a 5% chance of extremely bad outcomes in the 2016 survey, 2% in 2019, 5% in 2022 and 5% in 2023. ⁶

In the 2022 survey, participants were specifically asked about the chances of existential catastrophe caused by future AI advances — and again, over half of researchers thought the chances of an existential catastrophe was greater than 5%.⁷

So experts disagree on the degree to which AI poses an existential risk — a kind of threat we’ve argued deserves serious moral weight.

This fits with our understanding of the state of the research field. Three of the leading companiess developing AI — DeepMind, Anthropic and OpenAI — also have teams dedicated to figuring out how to solve technical safety issues that we believe could, for reasons we discuss at length below, lead to an existential threat to humanity.⁸

There are also several academic research groups (including at MIT, Cambridge, Carnegie Mellon University, and UC Berkeley) focusing on these same technical AI safety problems.⁹

It’s hard to know exactly what to take from all this, but we’re confident that it’s not a fringe position in the field to think that there is a material risk of outcomes as bad as an existential catastrophe. Some experts in the field maintain, though, that the risks are overblown.

Still, why do we side with those who are more concerned? In short, it’s because there are arguments we’ve found persuasive that AI could pose such an existential threat — arguments we will go through step by step below.

It’s important to recognise that the fact that many experts recognise there’s a problem doesn’t mean that everything’s OK because the experts have got it covered. Overall, we think this problem remains highly neglected (more on this below), especially as billions of dollars a year are spent to make AI more advanced.¹⁰

2. We’re making advances in AI extremely quickly

Before we try to figure out what the future of AI might look like, it’s helpful to take a look at what AI can already do.

Modern AI techniques involve machine learning (ML): models that improve automatically through data input. The most common form of this technique used today is known as deep learning.

Probably the most well-known ML-based product is ChatGPT. OpenAI’s commercialisation system — where you can pay for a much more powerful version of the product — led to revenue of over $2 billion by the end of 2023, making OpenAI one of the fastest growing startups ever.

If you’ve used ChatGPT, you may have been a bit underwhelmed. After all — while it’s great at some tasks, like coding and data analysis — it makes lots of mistakes. (Though note that the paid version tends to perform better than the free version.)

But we shouldn’t expect the frontier of AI to remain at the level of ChatGPT. There has been huge progress in what can be achieved with ML in only the last few years. Here are a few examples (from less recent to more recent):

• AlphaStar, which can beat top professional players at StarCraft II (January 2019)
• MuZero, a single system that learned to win games of chess, shogi, and Go — without ever being told the rules (November 2019)
• GPT-f, which can solve some Maths Olympiad problems (September 2020)
• AlphaFold 2, a huge step forward in solving the long-perplexing protein-folding problem (July 2021)
• Gato, a single ML model capable of doing a huge number of different things (including playing Atari, captioning images, chatting, and stacking blocks with a real robot arm), deciding what it should output based on the context (May 2022)
• Midjourney V6 (December 2023), Stable Diffusion XL (July 2023), DALL-E 3 (August 2023) and Imagen 2 (December 2023), all of which are capable of generating high-quality images from written descriptions
• Sora (February 2024), a model from OpenAI that can create realistic video from text prompts
  *And large language models, such as GPT-4, Claude, and Gemini — which we’ve become so familiar with through chatbots — continue to surpass benchmarks on maths, code, general knowledge, and reasoning ability.¹²

If you’re anything like us, you found the complexity and breadth of the tasks these systems can carry out surprising.

And if the technology keeps advancing at this pace, it seems clear there will be major effects on society. At the very least, automating tasks makes carrying out those tasks cheaper. As a result, we may see rapid increases in economic growth (perhaps even to the level we saw during the Industrial Revolution).

If we’re able to partially or fully automate scientific advancement we may see more transformative changes to society and technology.¹³

That could be just the beginning. We may be able to get computers to eventually automate anything humans can do. This seems like it has to be possible — at least in principle. This is because it seems that, with enough power and complexity, a computer should be able to simulate the human brain. This would itself be a way of automating anything humans can do (if not the most efficient method of doing so).

And as we’ll see in the next section, there are some indications that extensive automation may well be possible through scaling up existing techniques.

Current trends show rapid progress in the capabilities of ML systems

There are three things that are crucial to building AI through machine learning:

1. Good algorithms (e.g. more efficient algorithms are better)
2. Data to train an algorithm
3. Enough computational power (known as compute) to do this training

Epoch is a team of scientists investigating trends in the development of advanced AI — in particular, how these three inputs are changing over time.

They found that the amount of compute used for training the largest AI models has been rising exponentially — doubling on average every six months since 2010.

That means the amount of computational power used to train our largest machine learning models has grown by over one billion times.

Epoch also looked at how much compute has been needed to train a neural network to have the same performance on ImageNet (a well-known test data set for computer vision).

They found that the amount of compute required for the same performance has been falling exponentially — halving every 10 months.

So since 2012, the amount of compute required for the same level of performance has fallen by over 10,000 times. Combined with the increased compute used for training, that’s a lot of growth.

Finally, they found that the size of the data sets used to train the largest language models has been doubling roughly once a year since 2010.

It’s hard to say whether these trends will continue, but they speak to incredible gains over the past decade in what it’s possible to do with machine learning.

Indeed, it looks like increasing the size of models (and the amount of compute used to train them) introduces ever more sophisticated behaviour. This is how things like GPT-4 are able to perform tasks they weren’t specifically trained for.

These observations have led to the scaling hypothesis: that we can simply build bigger and bigger neural networks, and as a result we will end up with more and more powerful artificial intelligence, and that this trend of increasing capabilities may increase to human-level AI and beyond.

If this is true, we can attempt to predict how the capabilities of AI technology will increase over time simply by looking at how quickly we are increasing the amount of compute available to train models.

But as we’ll see, it’s not just the scaling hypothesis that suggests we could end up with extremely powerful AI relatively soon — other methods of predicting AI progress come to similar conclusions.

When can we expect transformative AI?

It’s difficult to predict exactly when we will develop AI that we expect to be hugely transformative for society (for better or for worse) — for example, by automating all human work or drastically changing the structure of society.¹⁴ But here we’ll go through a few approaches.

One option is to survey experts. Data from the 2023 survey of 3000 AI experts implies there is 33% probability of human-level machine intelligence (which would plausibly be transformative in this sense) by 2036, 50% probability by 2047, and 80% by 2100.¹⁵ There are a lot of reasons to be suspicious of these estimates,⁴ but we take it as one data point.

Ajeya Cotra (a researcher at Open Philanthropy) attempted to forecast transformative AI by comparing modern deep learning to the human brain. Deep learning involves using a huge amount of compute to train a model, before that model is able to perform some task. There’s also a relationship between the amount of compute used to train a model and the amount used by the model when it’s run. And — if the scaling hypothesis is true — we should expect the performance of a model to predictably improve as the computational power used increases. So Cotra used a variety of approaches (including, for example, estimating how much compute the human brain uses on a variety of tasks) to estimate how much compute might be needed to train a model that, when run, could carry out the hardest tasks humans can do. She then estimated when using that much compute would be affordable.

Cotra’s 2022 update on her report’s conclusions estimates that there is a 35% probability of transformative AI by 2036, 50% by 2040, and 60% by 2050 — noting that these guesses are not stable.¹⁶

Tom Davidson (also a researcher at Open Philanthropy) wrote a report to complement Cotra’s work. He attempted to figure out when we might expect to see transformative AI based only on looking at various types of research that transformative AI might be like (e.g. developing technology that’s the ultimate goal of a STEM field, or proving difficult mathematical conjectures), and how long it’s taken for each of these kinds of research to be completed in the past, given some quantity of research funding and effort.

Davidson’s report estimates that, solely on this information, you’d think that there was an 8% chance of transformative AI by 2036, 13% by 2060, and 20% by 2100. However, Davidson doesn’t consider the actual ways in which AI has progressed since research started in the 1950s, and notes that it seems likely that the amount of effort we put into AI research will increase as AI becomes increasingly relevant to our economy. As a result, Davidson expects these numbers to be underestimates.

Holden Karnofsky, co-CEO of Open Philanthropy, attempted to sum up the findings of others’ forecasts. He guessed in 2021 there was more than a 10% chance we’d see transformative AI by 2036, 50% by 2060, and 66% by 2100. And these guesses might be conservative, since they didn’t incorporate what we see as faster-than-expected progress since the earlier estimates were made.

All in all, AI seems to be advancing rapidly. More money and talent is going into the field every year, and models are getting bigger and more efficient.

Even if AI were advancing more slowly, we’d be concerned about it — most of the arguments about the risks from AI (that we’ll get to below) do not depend on this rapid progress.

However, the speed of these recent advances increases the urgency of the issue.

(It’s totally possible that these estimates are wrong – below, we discuss how the possibility that we might have a lot of time to work on this problem is one of the best arguments against this problem being pressing).

3. Power-seeking AI could pose an existential threat to humanity

We’ve argued so far that we expect AI to be an important — and potentially transformative — new technology.

We’ve also seen reason to think that such transformative AI systems could be built this century.

Now we’ll turn to the core question: why do we think this matters so much?

There could be a lot of reasons. If advanced AI is as transformative as it seems like it’ll be, there will be many important consequences. But here we are going to explain the issue that seems most concerning to us: AI systems could pose risks by seeking and gaining power.

We’ll argue that:

1. It’s likely that we’ll build AI systems that can make and execute plans to achieve goals
2. Advanced planning systems could easily be ‘misaligned’ — in a way that could lead them to make plans that involve disempowering humanity
3. Disempowerment by AI systems would be an existential catastrophe
4. People might deploy AI systems that are misaligned, despite this risk

Thinking through each step, I think there’s something like a 1% chance of an existential catastrophe resulting from power-seeking AI systems this century. This is my all things considered guess at the risk incorporating considerations of the argument in favour of the risk (which is itself probabilistic), as well as reasons why this argument might be wrong (some of which I discuss below). This puts me on the less worried end of 80,000 Hours staff, whose views on our last staff survey ranged from 1–55%, with a median of 15%.

It’s likely we’ll build advanced planning systems

We’re going to argue that future systems with the following three properties might pose a particularly important threat to humanity:¹⁷

1. They have goals and are good at making plans.

   Not all AI systems have goals or make plans to achieve those goals. But some systems (like some chess-playing AI systems) can be thought of in this way. When discussing power-seeking AI, we’re considering planning systems that are relatively advanced, with plans that are in pursuit of some goal(s), and that are capable of carrying out those plans.

2. They have excellent strategic awareness.

   A particularly good planning system would have a good enough understanding of the world to notice obstacles and opportunities that may help or hinder its plans, and respond to these accordingly. Following Carlsmith, we’ll call this strategic awareness, since it allows systems to strategise in a more sophisticated way.

3. They have highly advanced capabilities relative to today’s systems.

   For these systems to actually affect the world, we need them to not just make plans, but also be good at all the specific tasks required to execute those plans.

   Since we’re worried about systems attempting to take power from humanity, we are particularly concerned about AI systems that might be better than humans on one or more tasks that grant people significant power when carried out well in today’s world.

   For example, people who are very good at persuasion and/or manipulation are often able to gain power — so an AI being good at these things might also be able to gain power. Other examples might include hacking into other systems, tasks within scientific and engineering research, as well as business, military, or political strategy.

These systems seem technically possible and we’ll have strong incentives to build them

As we saw above, we’ve already produced systems that are very good at carrying out specific tasks.

We’ve also already produced rudimentary planning systems, like AlphaStar, which skilfully plays the strategy game Starcraft, and MuZero, which plays chess, shogi, and Go.¹⁸

We’re not sure whether these systems are producing plans in pursuit of goals per se, because we’re not sure exactly what it means to “have goals.” However, since they consistently plan in ways that achieve goals, it seems like they have goals in some sense.

Moreover, some existing systems seem to actually represent goals as part of their neural networks.¹⁹

That said, planning in the real world (instead of games) is much more complex, and to date we’re not aware of any unambiguous examples of goal-directed planning systems, or systems that exhibit high degrees of strategic awareness.

But as we’ve discussed, we expect to see further advances within this century. And we think these advances are likely to produce systems with all three of the above properties.

That’s because we think that there are particularly strong incentives (like profit) to develop these kinds of systems. In short: because being able to plan to achieve a goal, and execute that plan, seems like a particularly powerful and general way of affecting the world.

Getting things done — whether that’s a company selling products, a person buying a house, or a government developing policy — almost always seems to require these skills. One example would be assigning a powerful system a goal and expecting the system to achieve it — rather than having to guide it every step of the way. So planning systems seem likely to be (economically and politically) extremely useful.²⁰

And if systems are extremely useful, there are likely to be big incentives to build them. For example, an AI that could plan the actions of a company by being given the goal to increase its profits (that is, an AI CEO) would likely provide significant wealth for the people involved — a direct incentive to produce such an AI.

As a result, if we can build systems with these properties (and from what we know, it seems like we will be able to), it seems like we are likely to do so.²¹

Advanced planning systems could easily be dangerously ‘misaligned’

There are reasons to think that these kinds of advanced planning AI systems will be misaligned. That is, they will aim to do things that we don’t want them to do.²²

There are many reasons why systems might not be aiming to do exactly what we want them to do. For one thing, we don’t know how, using modern ML techniques, to give systems the precise goals we want (more here).²³

We’re going to focus specifically on some reasons why systems might by default be misaligned in such a way that they develop plans that pose risks to humanity’s ability to influence the world — even when we don’t want that influence to be lost.²⁴

What do we mean by “by default”? Essentially, unless we actively find solutions to some (potentially quite difficult) problems, then it seems like we’ll create dangerously misaligned AI. (There are reasons this might be wrong — which we discuss later.)

Three examples of “misalignment” in a variety of systems

It’s worth noting that misalignment isn’t a purely theoretical possibility (or specific to AI) — we see misaligned goals in humans and institutions all the time, and have also seen examples of misalignment in AI systems.²⁵

Example 1: Winning elections

The democratic political framework is intended to ensure that politicians make decisions that benefit society. But what political systems actually reward is winning elections, so that’s what many politicians end up aiming for.

This is a decent proxy goal — if you have a plan to improve people’s lives, they’re probably more likely to vote for you — but it isn’t perfect. As a result, politicians do things that aren’t clearly the best way of running a country, like raising taxes at the start of their term and cutting them right before elections.

That is to say, the things the system does are at least a little different from what we would, in a perfect world, want it to do: the system is misaligned.

Example 2: The profit incentive

Companies have profit-making incentives. By producing more, and therefore helping people obtain goods and services at cheaper prices, companies make more money.

This is sometimes a decent proxy for making the world better, but profit isn’t actually the same as the good of all of humanity (bold claim, we know). As a result, there are negative externalities: for example, companies will pollute to make money despite this being worse for society overall.

Again, we have a misaligned system, where the things the system does are at least a little different from what we would want it to do.

Example 3: Specification gaming in existing AI systems

DeepMind has documented examples of specification gaming: an AI doing well according to its specified reward function (which encodes our intentions for the system), but not doing what researchers intended.

In one example, a robot arm was asked to grasp a ball. But the reward was specified in terms of whether humans thought the robot had been successful. As a result, the arm learned to hover between the ball and the camera, fooling the humans into thinking that it had grasped the ball.²⁶

So we know it’s possible to create a misaligned AI system.

Why these systems could (by default) be dangerously misaligned

Here’s the core argument of this article. We’ll use all three properties from earlier: planning ability, strategic awareness, and advanced capabilities.

To start, we should realise that a planning system that has a goal will also develop ‘instrumental goals’: things that, if they occur, will make it easier to achieve an overall goal.

We use instrumental goals in plans all the time. For example, a high schooler planning their career might think that getting into university will be helpful for their future job prospects. In this case, “getting into university” would be an instrumental goal.

A sufficiently advanced AI planning system would also include instrumental goals in its overall plans.

If a planning AI system also has enough strategic awareness, it will be able to identify facts about the real world (including potential things that would be obstacles to any plans), and plan in light of them. Crucially, these facts would include that access to resources (e.g. money, compute, influence) and greater capabilities — that is, forms of power — open up new, more effective ways of achieving goals.

This means that, by default, advanced planning AI systems would have some worrying instrumental goals:

• Self-preservation — because a system is more likely to achieve its goals if it is still around to pursue them (in Stuart Russell’s memorable phrase, “You can’t fetch the coffee if you’re dead”).
• Preventing any changes to the AI system’s goals — since changing its goals would lead to outcomes that are different from those it would achieve with its current goals.
• Gaining power — for example, by getting more resources and greater capabilities.

Crucially, one clear way in which the AI can ensure that it will continue to exist (and not be turned off), and that its objectives will never be changed, would be to gain power over the humans who might affect it (we talk here about how AI systems might actually be able to do that).

What’s more, the AI systems we’re considering have advanced capabilities — meaning they can do one or more tasks that grant people significant power when carried out well in today’s world. With such advanced capabilities, these instrumental goals will not be out of reach, and as a result, it seems like the AI system would use its advanced capabilities to get power as part of the plan’s execution. If we don’t want the AI systems we create to take power away from us this would be a particularly dangerous form of misalignment.

In the most extreme scenarios, a planning AI system with sufficiently advanced capabilities could successfully disempower us completely.

As a (very non-rigorous) intuitive check on this argument, let’s try to apply it to humans.

Humans have a variety of goals. For many of these goals, some form of power-seeking is advantageous: though not everyone seeks power, many people do (in the form of wealth or social or political status), because it’s useful for getting what they want. This is not catastrophic (usually!) because, as human beings:

• We generally feel bound by human norms and morality (even people who really want wealth usually aren’t willing to kill to get it).
• We aren’t that much more capable or intelligent than one another. So even in cases where people aren’t held back by morality, they’re not able to take over the world.

(We discuss whether humans are truly power-seeking later.)

A sufficiently advanced AI wouldn’t have those limitations.

It might be hard to find ways to prevent this sort of misalignment

The point of all this isn’t to say that any advanced planning AI system will necessarily attempt to seek power. Instead, it’s to point out that, unless we find a way to design systems that don’t have this flaw, we’ll face significant risk.

It seems more than plausible that we could create an AI system that isn’t misaligned in this way, and thereby prevent any disempowerment. Here are some strategies we might take (plus, unfortunately, some reasons why they might be difficult in practice):²⁷

• Control the objectives of the AI system. We may be able to design systems that simply don’t have objectives to which the above argument applies — and thus don’t incentivise power-seeking behaviour. For example, we could find ways to explicitly instruct AI systems not to harm humans, or find ways to reward AI systems (in training environments) for not engaging in specific kinds of power-seeking behaviour (and also find ways to ensure that this behaviour continues outside the training environment).

  Carlsmith gives two reasons why doing this seems particularly hard.

  First, for modern ML systems, we don’t get to explicitly state a system’s objectives — instead we reward (or punish) a system in a training environment so that it learns on its own. This raises a number of difficulties, one of which is goal misgeneralisation. Researchers have uncovered real examples of systems that appear to have learned to pursue a goal in the training environment, but then fail to generalise that goal when they operate in a new environment. This raises the possibility that we could think we’ve successfully trained an AI system not to seek power — but that the system would seek power anyway when deployed in the real world.²⁸

  Second, when we specify a goal to an AI system (or, when we can’t explicitly do that, when we find ways to reward or punish a system during training), we usually do this by giving the system a proxy by which outcomes can be measured (e.g. positive human feedback on a system’s achievement). But often those proxies don’t quite work.²⁹ In general, we might expect that even if a proxy appears to correlate well with successful outcomes, it might not do so when that proxy is optimised for. (The examples above of politicians, companies, and the robot arm failing to grasp a ball are illustrations of this.) We’ll look at a more specific example of how problems with proxies could lead to an existential catastrophe here.

  For more on the specific difficulty of controlling the objectives given to deep neural networks trained using self-supervised learning and reinforcement learning, we recommend OpenAI governance researcher Richard Ngo’s discussion of how realistic training processes lead to the development of misaligned goals.

• Control the inputs into the AI system. AI systems will only develop plans to seek power if they have enough information about the world to realise that seeking power is indeed a way to achieve its goals.

• Control the capabilities of the AI system. AI systems will likely only be able to carry out plans to seek power if they have sufficiently advanced capabilities in skills that grant people significant power in today’s world.

But to make any strategy work, it will need to both:

• Retain the usefulness of the AI systems — and so remain economically competitive with less safe systems. Controlling the inputs and capabilities of AI systems will clearly have costs, so it seems hard to ensure that these controls, even if they’re developed, are actually used. But this is also a problem for controlling a system’s objectives. For example, we may be able to prevent power-seeking behaviour by ensuring that AI systems stop to check in with humans about any decisions they make. But these systems might be significantly slower and less immediately useful to people than systems that don’t stop to carry out these checks. As a result, there might still be incentives to use a faster, more initially effective misaligned system (we’ll look at incentives more in the next section).

• Continue to work as the planning ability and strategic awareness of systems improve over time. Some seemingly simple solutions (for example, trying to give a system a long list of things it isn’t allowed to do, like stealing money or physically harming humans) break down as the planning abilities of the systems increase. This is because, the more capable a system is at developing plans, the more likely it is to identify loopholes or failures in the safety strategy — and as a result, the more likely the system is to develop a plan that involves power-seeking.

Ultimately, by looking at the state of the research on this topic, and speaking to experts in the field, we think that there are currently no known ways of building aligned AI systems that seem likely to fulfil both these criteria.

So: that’s the core argument. There are many variants of this argument. Some have argued that AI systems might gradually shape our future via subtler forms of influence that nonetheless could amount to an existential catastrophe; others argue that the most likely form of disempowerment is in fact just killing everyone. We’re not sure how a catastrophe would be most likely to play out, but have tried to articulate the heart of the argument, as we see it: that AI presents an existential risk.

There are definitely reasons this argument might not be right! We go through some of the reasons that seem strongest to us below. But overall it seems possible that, for at least some kinds of advanced planning AI systems, it will be harder to build systems that don’t seek power in this dangerous way than to build systems that do.

Disempowerment by AI systems would be an existential catastrophe

When we say we’re concerned about existential catastrophes, we’re not just concerned about risks of extinction. This is because the source of our concern is rooted in longtermism: the idea that the lives of all future generations matter, and so it’s extremely important to protect their interests.

This means that any event that could prevent all future generations from living lives full of whatever you think makes life valuable (whether that’s happiness, justice, beauty, or general flourishing) counts as an existential catastrophe.

It seems extremely unlikely that we’d be able to regain power over a system that successfully disempowers humanity. And as a result, the entirety of the future — everything that happens for Earth-originating life, for the rest of time — would be determined by the goals of systems that, although built by us, are not aligned with us. Perhaps those goals will create a long and flourishing future, but we see little reason for confidence.³⁰

This isn’t to say that we don’t think AI also poses a risk of human extinction. Indeed, we think making humans extinct is one highly plausible way in which an AI system could completely and permanently ensure that we are never able to regain power.

People might deploy misaligned AI systems despite the risk

Surely no one would actually build or use a misaligned AI if they knew it could have such terrible consequences, right?

Unfortunately, there are at least two reasons people might create and then deploy misaligned AI — which we’ll go through one at a time:³¹

1. People might think it’s aligned when it’s not

Imagine there’s a group of researchers trying to tell, in a test environment, whether a system they’ve built is aligned. We’ve argued that an intelligent planning AI will want to improve its abilities to effect changes in pursuit of its objective, and it’s almost always easier to do that if it’s deployed in the real world, where a much wider range of actions are available. As a result, any misaligned AI that’s sophisticated enough will try to understand what the researchers want it to do and at least pretend to be doing that, deceiving the researchers into thinking it’s aligned. (For example, a reinforcement learning system might be rewarded for certain apparent behaviour during training, regardless of what it’s actually doing.)

Hopefully, we’ll be aware of this sort of behaviour and be able to detect it. But catching a sufficiently advanced AI in deception seems potentially harder than catching a human in a lie, which isn’t always easy. For example, a sufficiently intelligent deceptive AI system may be able to deceive us into thinking we’ve solved the problem of AI deception, even if we haven’t.

If AI systems are good at deception, and have sufficiently advanced capabilities, a reasonable strategy for such a system could be to deceive humans completely until the system has a way to guarantee it can overcome any resistance to its goals.

2. There are incentives to deploy systems sooner rather than later

We might also expect some people with the ability to deploy a misaligned AI to charge ahead despite any warning signs of misalignment that do come up, because of race dynamics — where people developing AI want to do so before anyone else.

For example, if you’re developing an AI to improve military or political strategy, it’s much more useful if none of your rivals have a similarly powerful AI.

These incentives apply even to people attempting to build an AI in the hopes of using it to make the world a better place.

For example, say you’ve spent years and years researching and developing a powerful AI system, and all you want is to use it to make the world a better place. Simplifying things a lot, say there are two possibilities:

1. This powerful AI will be aligned with your beneficent aims, and you’ll transform society in a potentially radically positive way.
2. The AI will be sufficiently misaligned that it’ll take power and permanently end humanity’s control over the future.

Let’s say you think there’s a 90% chance that you’ve succeeded in building an aligned AI. But technology often develops at similar speeds across society, so there’s a good chance that someone else will soon also develop a powerful AI. And you think they’re less cautious, or less altruistic, so you think their AI will only have an 80% chance of being aligned with good goals, and pose a 20% chance of existential catastrophe. And only if you get there first can your more beneficial AI be dominant. As a result, you might decide to go ahead with deploying your AI, accepting the 10% risk.

4. Even if we find a way to avoid power-seeking, there are still risks

So far we’ve described what a large proportion of researchers in the field² think is the major existential risk from potential advances in AI, which depends crucially on an AI seeking power to achieve its goals.

If we can prevent power-seeking behaviour, we will have reduced existential risk substantially.

But even if we succeed, there are still existential risks that AI could pose.

There are at least two ways these risks could arise:

• We expect that AI systems will help increase the rate of scientific progress.³² While there would be clear benefits to this automation — the rapid development of new medicine, for example — some forms of technological development can pose threats, including existential threats, to humanity. This technological advancement might increase our available destructive power or make dangerous technologies cheaper or more widely accessible.
• We might start to see AI automate many – or possibly even all – economically important tasks. It’s hard to predict exactly what the effects of this would be on society. But it seems plausible that this could increase existential risks. For example, if AI systems are highly transformative, then their use (or potential use) could possibly create insurmountable power imbalances. Even the threat of this might be enough. For example, a military might feel pushed to create transformative automated weapons because it knows or believes its enemies are doing so, even if this dynamic benefits no one.

We know of several specific areas in which advanced AI may increase existential risks, though are likely others we haven’t thought of.

Bioweapons

In 2022, Collaborations Pharmaceuticals — a small research corporation in North Carolina — were building an AI model to help determine the structure of new drugs. As part of this process, they trained the model to penalise drugs that it predicted were toxic. This had just one problem: you could run the toxicity prediction in reverse to invent new toxic drugs.

Some of the deadliest events in human history have been pandemics. Pathogens’ ability to infect, replicate, kill, and spread — often undetected — make them exceptionally dangerous.

Even without AI, advancing biotechnology poses extreme risks. It potentially provides opportunities for state actors or terrorists to create mass-casualty events.

Advances in AI have the potential to make biotechnology more dangerous.

For example:

1. Dual-use tools, like the automation of laboratory processes, could lower the barriers for rogue actors trying to manufacture a dangerous pandemic virus.³³ The Collaborations Pharmaceuticals model is an example of a dual-use tool (although it’s not particularly dangerous).

2. AI-based biological design tools could enable sophisticated actors to reprogram the genomes of dangerous pathogens to specifically enhance their lethality, transmissibility, and immune evasion.³⁴

If AI is able to advance the rate of scientific and technological progress, these risks may be amplified and accelerated — making dangerous technology more widely available or increasing its possible destructive power.³⁵

In the 2023 survey of AI experts, 73% of respondents said they had either “extreme” or “substantial” concern that in the future Al will let “dangerous groups make powerful tools (e.g. engineered viruses).”³⁶

Intentionally dangerous AI agents

Most of this article discusses the risk of power-seeking AI systems that arise unintentionally due to misalignment.

But we can’t rule out the possibility that some people might intentionally create rogue AI agents that seek to disempower humanity. It might seem hard to imagine, but extremist ideologies of many forms have inspired humans to carry out radically violent and even self-destructive plans.³⁷

Cyberweapons

AI can already be used in cyberattacks, such as phishing, and more powerful AI may cause greater information security challenges (though it could also be useful in cyberdefense).

On its own, AI-enabled cyberwarfare is unlikely to pose an existential threat to humanity. Even the most damaging and costly societal-scale cyberattacks wouldn’t approach an extinction-level event.

But AI-enabled cyberattacks could provide access to other dangerous technology, such as bioweapons, nuclear arsenals, or autonomous weapons. So there may be genuine existential risks posed by AI-related cyberweapons, but they will most likely run through another existential risk.

The cyber capabilities of AI systems are also relevant to how a power-seeking AI could actually take power.

Other dangerous tech

If AI systems generally accelerate the rate of scientific and technological progress, we think it’s reasonably likely that we’ll invent new dangerous technologies.

For example, atomically precise manufacturing, sometimes called nanotechnology, has been hypothesised as an existential threat — and it’s a scientifically plausible technology that AI could help us invent far sooner than we would otherwise.

In The Precipice, Toby Ord estimated the chances of an existential catastrophe by 2120 from “unforeseen anthropogenic risks” at 1 in 30. This estimate suggests there could be other discoveries, perhaps involving yet to be understood physics, that could enable the creation of technologies with catastrophic consequences.³⁸

AI could empower totalitarian governments

An AI-enabled authoritarian government could completely automate the monitoring and repression of its citizens, as well as significantly influence the information people see, perhaps making it impossible to coordinate action against such a regime.

AI is already facilitating the ability of governments to monitor their own citizens.

The NSA is using AI to help filter the huge amounts of data they collect, significantly speeding up their ability to identify and predict the actions of people they are monitoring. In China, AI is increasingly being used for facial recognition and predictive policing, including automated racial profiling and automatic alarms when people classified as potential threats enter certain public places.

These sorts of surveillance technologies seem likely to significantly improve — thereby increasing governments’ abilities to control their populations.

At some point, authoritarian governments could extensively use AI-related technology to:

• Monitor and track dissidents
• Preemptively suppress opposition to the ruling party
• Control the military and dominate external actors
• Manipulate information flows and carefully shape public opinion

Again, in the 2023 survey of AI experts, 73% of respondents expressed “extreme” or “substantial” concern that in the future authoritarian rulers could “use Al to control their population.”³⁶

If a regime achieved a form of truly stable totalitarianism, it could make people’s lives much worse for a long time into the future, making it a particularly scary possible scenario resulting from AI. (Read more in our article on risks of stable totalitarianism.)

AI could worsen war

We’re concerned that great power conflict could also pose a substantial threat to our world, and advances in AI seem likely to change the nature of war — through lethal autonomous weapons³⁹ or through automated decision making.⁴⁰

In some cases, great power war could pose an existential threat — for example, if the conflict is nuclear. Some argue that lethal autonomous weapons, if sufficiently powerful and mass-produced, could themselves constitute a new form of weapon of mass destruction.

And if a single actor produces particularly powerful AI systems, this could be seen as giving them a decisive strategic advantage. Such an outcome, or even the expectation of such an outcome, could be highly destabilising.

Imagine that the US was working to produce a planning AI that’s intelligent enough to ensure that Russia or China could never successfully launch another nuclear weapon. This could incentivise a first strike from the actor’s rivals before these AI-developed plans can ever be put into action.

This is because nuclear deterrence can benefit from symmetry between the abilities of nuclear powers, in that the threat of a nuclear response to a first strike is believable and therefore a deterrent to a first strike. Advances in AI, which could be directly applied to nuclear forces, could create asymmetries in the capabilities of nuclear-armed nations. This could include improving early warning systems, air defence systems, and cyberattacks that disable weapons.

For example, many countries use submarine-launched ballistic missiles as part of their nuclear deterrence systems — the idea is that if nuclear weapons can be hidden under the ocean, they will never be destroyed in the first strike. This means that they can always be used for a counterattack, and therefore act as an effective deterrent against first strikes. But AI could make it far easier to detect submarines underwater, enabling their destruction in a first strike — removing this deterrent.

Many other destabilising scenarios are likely possible.

A report from the Stockholm International Peace Research Institute found that, while AI could potentially also have stabilising effects (for example by making everyone feel more vulnerable, decreasing the chances of escalation), destabilising effects could arise even before advances in AI are actually deployed. This is because one state’s belief that their opponents have new nuclear capabilities can be enough to disrupt the delicate balance of deterrence.

Luckily, there are also plausible ways in which AI could help prevent the use of nuclear weapons — for example, by improving the ability of states to detect nuclear launches, which would reduce the chances of false alarms like those that nearly caused nuclear war in 1983.

Overall, we’re uncertain about whether AI will substantially increase the risk of nuclear or conventional conflict in the short term — it could even end up decreasing the risk. But we think it’s important to pay attention to possible catastrophic outcomes and take reasonable steps to reduce their likelihood.

Other risks from AI

We’re also concerned about the following issues:

• Existential threats that result not from the power-seeking behaviour of AI systems, but from the interaction between AI systems. (In order to pose a risk, these systems would still need to be, to some extent, misaligned.)
• Other ways we haven’t thought of that AI systems could be misused — especially ones that might significantly affect future generations.
• Other moral mistakes made in the design and use of AI systems, particularly if future AI systems are themselves deserving of moral consideration. For example, we might (inadvertently) create sentient AI systems, which could then suffer in huge numbers. We think this could be extremely important, so we’ve written about it in a separate problem profile.

So, how likely is an AI-related catastrophe?

This is a really difficult question to answer.

There are no past examples we can use to determine the frequency of AI-related catastrophes.

All we have to go off are arguments (like the ones we’ve given above), and less relevant data like the history of technological advances. And we’re definitely not certain that the arguments we’ve presented are completely correct.

Consider the argument we gave earlier about the dangers of power-seeking AI in particular, based off Carlsmith’s report. At the end of his report, Carlsmith gives some rough guesses of the chances that each stage of his argument is correct (conditional on the previous stage being correct):

1. By 2070 it will be possible and financially feasible to build strategically aware systems that can outperform humans on many power-granting tasks, and that can successfully make and carry out plans: Carlsmith guesses there’s a 65% chance of this being true.
2. Given this feasibility, there will be strong incentives to build such systems: 80%.
3. Given both the feasibility and incentives to build such systems, it will be much harder to develop aligned systems that don’t seek power than to develop misaligned systems that do, but which are at least superficially attractive to deploy: 40%.
4. Given all of this, some deployed systems will seek power in a misaligned way that causes over $1 trillion (in 2021 dollars) of damage: 65%.
5. Given all the previous premises, misaligned power-seeking AI systems will end up disempowering basically all of humanity: 40%.
6. Given all the previous premises, this disempowerment will constitute an existential catastrophe: 95%.

Multiplying these numbers together, Carlsmith estimated that there’s a 5% chance that his argument is right and there will be an existential catastrophe from misaligned power-seeking AI by 2070. When we spoke to Carlsmith, he noted that in the year between the writing of his report and the publication of this article, his overall guess at the chance of an existential catastrophe from power-seeking AI by 2070 had increased to >10%.⁴¹

The overall probability of existential catastrophe from AI would, in Carlsmith’s view, be higher than this, because there are other routes to possible catastrophe — like those discussed in the previous section — although our guess is that these other routes are probably a lot less likely to lead to existential catastrophe.

For another estimate, in The Precipice, philosopher and advisor to 80,000 Hours Toby Ord estimated a 1-in-6 risk of existential catastrophe by 2120 (from any cause), and that 60% of this risk comes from misaligned AI — giving a total of a 10% risk of existential catastrophe from misaligned AI by 2120.

A 2021 survey of 44 researchers working on reducing existential risks from AI found the median risk estimate was 32.5% — the highest answer given was 98%, and the lowest was 2%.⁴² There’s obviously a lot of selection bias here: people choose to work on reducing risks from AI because they think this is unusually important, so we should expect estimates from this survey to be substantially higher than estimates from other sources. But there’s clearly significant uncertainty about how big this risk is, and huge variation in answers.

All these numbers are shockingly, disturbingly high. We’re far from certain that all the arguments are correct. But these are generally the highest guesses for the level of existential risk of any of the issues we’ve examined (like engineered pandemics, great power conflict, climate change, or nuclear war).

That said, I think there are reasons why it’s harder to make guesses about the risks from AI than other risks – and possibly reasons to think that the estimates we’ve quoted above are systematically too high.

If I was forced to put a number on it, I’d say something like 1%. This number includes considerations both in favour and against the argument. I’m less worried than other 80,000 Hours staff — our position as an organisation is that the risk is between 3% and 50%.

All this said, the arguments for such high estimates of the existential risk posed by AI are persuasive — making risks from AI a top contender for the most pressing problem facing humanity.

5. We can tackle these risks

We think one of the most important things you can do would be to help reduce the gravest risks that AI poses.

This isn’t just because we think these risks are high — it’s also because we think there are real things we can do to reduce these risks.

We know of two main ways people work to reduce these risks:

1. Technical AI safety research
2. AI governance and policy work

There are lots of ways to contribute to this work. In this section, we discuss many broad approaches within both categories to illustrate the point that there are things we can do to address these risks. Below, we discuss the kinds of careers you can pursue to work on these kinds of approaches.

Technical AI safety research

The benefits of transformative AI could be huge, and there are many different actors involved (operating in different countries), which means it will likely be really hard to prevent its development altogether.

(It’s also possible that it wouldn’t even be a good idea if we could — after all, that would mean forgoing the benefits as well as preventing the risks.)

As a result, we think it makes more sense to focus on making sure that this development is safe — meaning that it has a high probability of avoiding all the catastrophic failures listed above.

One way to do this is to try to develop technical solutions to prevent the kind of power-seeking behaviour we discussed earlier — this is generally known as working on technical AI safety, sometimes called just “AI safety” for short.

We discuss this path in more detail here:

Career review of technical AI safety research

Approaches

There are lots of approaches to technical AI safety, including:

• Scalably learning from human feedback. Examples include iterated amplification, AI safety via debate, building AI assistants that are uncertain about our goals and learn them by interacting with us, and other ways to get AI systems to report truthfully what they know.
• Threat modelling. An example of this work would be demonstrating the possibility of (allowing us to study) dangerous capabilities, like deceptive or manipulative AI systems. This approach splits into work that evaluates whether a model has dangerous capabilities (like the work of METR in evaluating GPT-4), and work that evaluates whether a model would cause harm in practice (like Anthropic’s research into the behaviour of large language models and this paper on goal misgeneralisation). It can also include work to find ‘model organisms of misalignment’, in the hope of better understanding their dangers.
• Work to figure out how to control powerful AI systems – preventing them from causing harm even if they are unsafe. Read more in this blogpost from the team at Redwood Research working on control.
• Interpretability research. This work involves studying why AI systems do what they do and trying to put it into human-understandable terms. For example, this paper examined how AlphaZero learns chess, and this paper looked into finding latent knowledge in language models without supervision. This category also includes mechanistic interpretability — for example, Zoom In: An Introduction to Circuits by Olah et al.. For more, see this survey paper, as well as Hubinger’s a transparency and interpretability tech tree, and Nanda’s A Longlist of Theories of Impact for Interpretability for overviews of of how interpretability research could reduce existential risk from AI.
• Other anti-misuse research to reduce the risks of catastrophe caused by misuse of systems. For example, this work includes training AIs so they’re hard to use for dangerous purposes. (Note there’s lots of overlap with the other work on this list).
• Research to increase the robustness of neural networks. This work involves ensuring that the sorts of behaviour neural networks display when exposed to one set of inputs continues when exposed to inputs they haven’t previously been exposed to, in order to prevent AI systems changing to unsafe behaviour. See section 2 of Unsolved Problems in AI safety for more.
• Work to build cooperative AI. Find ways to ensure that even if individual AI systems seem safe, they don’t produce bad outcomes through interacting with other sociotechnical systems. For more, see Open Problems in Cooperative AI by Dafoe et al. or the Cooperative AI Foundation. This seems particularly relevant for the reduction of ‘s-risks.’
• More generally, there are some unified safety plans. For more, see Hubinger’s 11 possible proposals for building safe advanced AI, or Karnofsky’s How might we align transformative AI if it’s developed very soon.⁴³

See Neel Nanda’s overview of the AI alignment landscape for more details.

Read more about technical AI safety research below.

AI governance and policy

Reducing the gravest risks from AI will require sound high-level decision making and policy, both at AI companies themselves and in governments.

As AI has advanced and drawn increasing interest from customers and investors, governments have shown an interest in regulating the technology. Some have already taken significant steps to play a role in managing the development of AI, including:

• The US and the UK have each established their own national AI Safety Institutes.
• The European Union has passed the EU AI Act, which contains specific provisions for governing general-purpose AI models that pose systemic risks.
• The UK and then South Korea have hosted the first two AI Safety Summits (in 2023 and 2024), a series of high-profile summits aiming to coordinate between different countries, academics, researchers, and civil society leaders.
• China has implemented regulations targeting recommendation algorithms, synthetic AI content, generative AI models, and facial recognition technology.
• The US instituted export controls to reduce China’s access to the most cutting-edge chips used in AI development.

Much more will need to be done to reduce the biggest risks — including continuous evaluation of the AI governance landscape to assess overall progress.

We discuss this career path in more detail here:

Career review of AI strategy and policy careers

Approaches

People working in AI policy have proposed a range of approaches to reducing risk as AI systems get more powerful.

We don’t necessarily endorse all the ideas below, but what follows is a list of some prominent policy approaches that could be aimed at reducing the largest dangers from AI:⁴⁴

• Responsible scaling policies: some major AI companies have already begun developing internal frameworks for assessing safety as they scale up the size and capabilities of their systems. These frameworks introduce safeguards that are intended to become increasingly stringent as AI systems become more potentially dangerous, and ensure that AI systems’ capabilities don’t outpace companies’ abilities to keep systems safe. Many argue that these internal policies are not sufficient for safety, but they may represent a promising step for reducing risk. You can see versions of such policies from Anthropic, Google DeepMind, and OpenAI.
• Standards and evaluation: governments may also develop industry-wide benchmarks and testing protocols to assess whether AI systems pose major risks. The non-profit METR and the UK AI Safety Institute are among the organisations currently developing these evaluations to test AI models before and after they are released. This can include creating standardised metrics for an AI systems’s capabilities and potential to cause harm, as well as propensity for power-seeking or misalignment.
• Safety cases: this practice involves requiring AI developers to provide comprehensive documentation demonstrating the safety and reliability of their systems before deployment. This approach is similar to safety cases used in other high-risk industries like aviation or nuclear power.⁴⁵ You can see discussion of this idea in a paper from Clymer et al and in a post from Geoffrey Irving at the UK AI Safety Institute.
• Information security standards: we can establish robust rules for protecting AI-related data, algorithms, and infrastructure from unauthorised access or manipulation — particularly the AI model weights. Rand released a detailed report analysing the security risks to major AI companies, particularly from state actors.
• Liability law: existing law already imposes some liability on companies that create dangerous products or cause significant harm to the public, but its application to AI models and risk in particular is unclear. Clarifying how liability applies to companies that create dangerous AI models could incentivise them to take additional steps to reduce risk. Law professor Gabriel Weil has written about this idea.
• Compute governance: governments may regulate access to and use of high-performance computing resources necessary for training large AI models. The US restrictions on exporting state-of-the-art chips to China is one example of such a policy, and others are possible. Companies could also be required to install hardware-level safety features directly into AI chips or processors. These could be used to track chips and verify they’re not in the possession of anyone who shouldn’t have them, or for other purposes. You can learn more about this topic in our interview with Lennart Heim and in this report from the Center for a New American Security.
• International coordination: Fostering global cooperation on AI governance to ensure consistent standards. This could involve treaties, international organisations, or multilateral agreements on AI development and deployment. We discuss some related considerations in our article on China-related AI safety and governance paths.
• Societal adaptation: it may be critically important to prepare society for the widespread integration of AI and the potential risks it poses. For example, we might need to develop new information security measures to protect crucial data in a world with AI-enabled hacking. Or we may want to implement strong controls to prevent handing over key societal decisions to AI systems.⁴⁶
• Pausing scaling if appropriate: some argue that we should currently pause all scaling of larger AI models because of the dangers the technology poses. We have featured some discussion of this idea on our podcast, and it seems hard to know when or if this would be a good idea. If carried out, it could involve industry-wide agreements or regulatory mandates to pause scaling efforts when necessary.

The details, benefits, and downsides of many of these ideas have yet to be fully worked out, so it’s crucial that we do more research. And this list isn’t comprehensive — there are likely other important policy interventions and governance strategies worth pursuing.

We also need more forecasting research into what we should expect to happen with AI, such as the work done at Epoch AI.

6. This work is neglected

In 2022, we estimated there were around 400 people around the world working directly on reducing the chances of an AI-related existential catastrophe (with a 90% confidence interval ranging between 200 and 1,000). Of these, about three quarters worked on technical AI safety research, with the rest split between strategy (and other governance) research and advocacy.⁴⁷ We also estimated that there were around 800 people working in complementary roles, but we’re highly uncertain about this figure.⁴⁸

In The Precipice, Ord estimated that there was between $10 million and $50 million spent on reducing AI risk in 2020.

That might sound like a lot of money, but we’re spending something like 1,000 times that amount¹⁰ on speeding up the development of transformative AI via commercial capabilities research and engineering at large AI companies.

To compare the $50 million spent on AI safety in 2020 to other well-known risks, we’re currently spending several hundreds of billions per year on tackling climate change.

Because this field is so neglected and has such high stakes, we think your impact working on risks from AI could be much higher than working on many other areas — which is why our top two recommended career paths for making a big positive difference in the world are technical AI safety and AI policy research and implementation.

What do we think are the best arguments against this problem being pressing?

As we said above, we’re not totally sure the arguments we’ve presented for AI representing an existential threat are right. Though we do still think that the chance of catastrophe from AI is high enough to warrant many more people pursuing careers to try to prevent such an outcome, we also want to be honest about the arguments against doing so, so you can more easily make your own call on the question.

Here we’ll cover the strongest reasons (in our opinion) to think this problem isn’t particularly pressing. In the next section we’ll cover some common objections that (in our opinion) hold up less well, and explain why.

Arguments against working on AI risk to which we think there are strong responses

We’ve just discussed the major objections to working on AI risk that we think are most persuasive. In this section, we’ll look at objections that we think are less persuasive, and give some reasons why.

What you can do concretely to help

As we mentioned above, we know of two main ways to help reduce existential risks from AI:

1. Technical AI safety research
2. AI governance and policy work

The biggest way you could help would be to pursue a career in either one of these areas, or in a supporting area.

The first step is learning a lot more about the technologies, problems, and possible solutions. We’ve collated some lists of our favourite resources here, and our top recommendation is to take a look at the technical alignment curriculum from AGI Safety Fundamentals.

Technical AI safety

If you’re interested in a career in technical AI safety, the best place to start is our career review of being an AI safety researcher.

If you want to learn more about technical AI safety as a field of research — e.g. the different techniques, schools of thought, and threat models — our top recommendation is to take a look at the technical alignment curriculum from AGI Safety Fundamentals.

It’s important to note that you don’t have to be an academic or an expert in AI or AI safety to contribute to AI safety research. For example, software engineers are needed at many places conducting technical safety research, and we also highlight more roles below.

You can see a list of key organisations where you might do this kind of work in the full career review.

AI governance and policy work

If you’re interested in a career in AI governance and policy, the best place to start is our AI governance and policy career review.

You don’t need to be a bureaucrat in a grey suit to have a career in AI governance and policy — there are roles suitable for a wide range of skill sets. In particular, people with technical skills in machine learning and related fields are needed for governance work (although those skills are certainly not necessary).

We split this career path into six different kinds of roles:

1. Government roles
2. Research
3. Industry work
4. Advocacy and lobbying
5. Third-party auditing and evaluation
6. International work and coordination

We also have specific articles on working in US AI policy and China-related AI safety and governance paths.

And you can learn more about where specifically you might work in this career path in our career review.

If you’re new to the topic and interested in learning more broadly about AI governance, our top recommendation is to take a look at the governance curriculum from AGI safety fundamentals.

Complementary (yet crucial) roles

Even in a research organisation, around half of the staff will be doing other tasks essential for the organisation to perform at its best and have an impact. Having high-performing people in these roles is crucial.

We think the importance of these roles is often underrated because the work is less visible. So we’ve written several career reviews on these areas to help more people enter these careers and succeed, including:

• Operations management to help impactful organisations grow and function as effectively as possible.
• Research management at an AI safety research organisation.
• Being an executive assistant to someone who’s doing really important work on safety and governance.

Other ways to help

AI safety is a big problem and it needs help from people doing a lot of different kinds of work.

One major way to help is to work in a role that directs funding or people towards AI risk, rather than working on the problem directly. We’ve reviewed a few career paths along these lines, including:

• Founding new projects — in this case, starting new initiatives aimed at reducing risks from advanced AI.
• Being a grantmaker to fund promising projects focused on reducing catastrophic AI risk.
• Working in communication roles.
• Helping to build communities of people working on this problem. The most relevant community is the AI safety community itself, but it could also be impactful to help build the community of people working on the world’s most pressing problems (including risks from AI).

There are ways all of these could go wrong, so the first step is to become well-informed about the issue.

There are also other technical roles besides safety research that could help contribute, like:

• Working in information security to protect AI (or the results of key experiments) from misuse, theft, or tampering.
• Becoming an expert in AI hardware as a way of steering AI progress in safer directions.

You can read about all these careers — why we think they’re helpful, how to enter them, and how you can predict whether they’re a good fit for you — on our career reviews page.

Find vacancies on our job board

Our job board features opportunities in AI technical safety and governance:

View all opportunities

Top resources to learn more

We've hit you with a lot of further reading throughout this article — here are a few of our favourites:

• AI could defeat all of us combined and the "most important century" blog post series by Holden Karnofsky, co-CEO of Open Philanthropy, argues that the 21st century could be the most important century ever for humanity as a result of AI.
• Why AI alignment could be hard with modern deep learning by Open Philanthropy researcher Cotra is a gentle introduction to how risks from power-seeking AI could play out with current machine learning methods. Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover, also by Cotra, provides a much more detailed description of how risks could play out (which we'd recommend for people familiar with ML).
• AGI safety from first principles provides OpenAI governance researcher Richard Ngo's perspective on how to think about risks from artificial general intelligence.
• Is power-seeking AI an existential risk? by Open Philanthropy researcher Joseph Carlsmith is an in-depth look covering exactly how and why AI could cause the disempowerment of humanity (but watch out — it's even longer than this article!). It's also available as an audio narration. For a shorter summary, see Carlsmith's talk on the same topic.
• Distinguishing AI takeover scenarios by Sam Clarke and Sammy Martin summarises various ways in which AI could go wrong.
• AI governance: Opportunity and theory of impact by DeepMind governance lead Allan Dafoe explores ways in which research into AI governance could effect change.
• A bird's-eye view of the AI alignment landscape by Neel Nanda summarises the different ways in which technical alignment research could reduce the risk from AI.
• An overview of 11 proposals for building safe advanced AI by Evan Hubinger discusses and evaluates plausible techniques for AI alignment.
• Podcasts: The AI X-risk Research Podcast, particularly episode 12 with Paul Christiano and episode 13 with Richard Ngo — both of which serve as excellent introductions to AI risk.

On The 80,000 Hours Podcast, we have a number of in-depth interviews with people actively working to positively shape the development of artificial intelligence:

• Paul Christiano on how OpenAI is developing real solutions to the 'AI alignment problem', and his vision of how humanity will progressively hand over decision-making to AI systems
• Allan Dafoe on trying to prepare the world for the possibility that AI will destabilise global politics
• Carl Shulman on the economy and national security and government and society after AGI
• Richard Ngo on large language models, OpenAI, and striving to make the future go well
• Ajeya Cotra on accidentally teaching AI models to deceive us
• Jan Leike on how to become a machine learning alignment researcher (from 2018) and OpenAI's massive push to make superintelligence safe in 4 years or less (from 2023)
• Nathan Labenz on the final push for AGI, understanding OpenAI's leadership drama, and red-teaming frontier models
• Rohin Shah on DeepMind and trying to fairly hear out both AI doomers and doubters
• Tom Davidson on how quickly AI could transform the world
• Dario Amodei on OpenAI and how AI will change the world for good and ill
• Miles Brundage on the world's desperate need for AI strategists and policy experts
• Holden Karnofsky, cofounder of GiveWell and Open Philanthropy, has been on three of our podcasts, explaining:
  • How AIs might take over even if they're no smarter than humans, and his four-part playbook for AI risk
  • How philanthropy can have maximum impact by taking big risks (including a discussion of his work in positively shaping the development of AI)
  • Why this might be the most important century
• PhD or programming? Fast paths into aligning AI as a machine learning engineer, according to ML engineers Catherine Olsson & Daniel Ziegler

If you want to go into much more depth, the AGI safety fundamentals course is a good starting point. There are two tracks to choose from: technical alignment or AI governance. If you have a more technical background, you could try Intro to ML Safety, a course from the Center for AI Safety.

And finally, here are a few general sources (rather than specific articles) that you might want to explore:

• The AI Alignment Forum, which is aimed at researchers working in technical AI safety.
• AI Impacts, a project that aims to improve society's understanding of the likely impacts of human-level artificial intelligence.
• The Alignment Newsletter, a weekly publication with recent content relevant to AI alignment with thousands of subscribers.
• Import AI, a weekly newsletter about artificial intelligence by Jack Clark (cofounder of Anthropic), read by more than 10,000 experts.
• Jeff Ding's ChinAI Newsletter, weekly translations of writings from Chinese thinkers on China's AI landscape.

Read next:  Explore other pressing world problems

Want to learn more about global issues we think are especially pressing? See our list of issues that are large in scale, solvable, and neglected, according to our research.

Continue →

Acknowledgements

Huge thanks to Joel Becker, Tamay Besiroglu, Jungwon Byun, Joseph Carlsmith, Jesse Clifton, Emery Cooper, Ajeya Cotra, Andrew Critch, Anthony DiGiovanni, Noemi Dreksler, Ben Edelman, Lukas Finnveden, Emily Frizell, Ben Garfinkel, Katja Grace, Lewis Hammond, Jacob Hilton, Samuel Hilton, Michelle Hutchinson, Caroline Jeanmaire, Kuhan Jeyapragasan, Arden Koehler, Daniel Kokotajlo, Victoria Krakovna, Alex Lawsen, Howie Lempel, Eli Lifland, Katy Moore, Luke Muehlhauser, Neel Nanda, Linh Chi Nguyen, Luisa Rodriguez, Caspar Oesterheld, Ethan Perez, Charlie Rogers-Smith, Jack Ryan, Rohin Shah, Buck Shlegeris, Marlene Staib, Andreas Stuhlmüller, Luke Stebbing, Nate Thomas, Benjamin Todd, Stefan Torges, Michael Townsend, Chris van Merwijk, Hjalmar Wijk, and Mark Xu for either reviewing this article or their extremely thoughtful and helpful comments and conversations. (This isn’t to say that they would all agree with everything we’ve said here — in fact, we’ve had many spirited disagreements in the comments on this article!)

The post Preventing an AI-related catastrophe appeared first on 80,000 Hours.

Here at 80,000 Hours, we’re often trying to find ways to protect future generations.

If we’d been trying to do that in 1950, one thing would have been at the top of everyone’s minds: the terrifying threat of nuclear annihilation. Indeed, many of the world’s greatest thinkers, politicians, and communicators devoted their careers to understanding and reducing the threat — people like Thomas Schelling, Carl Sagan and even, in his later years, Albert Einstein.

But since the end of the Cold War, the nuclear expert has all but disappeared.

And that’s a problem.

It’s a problem because the risk of nuclear war didn’t just disappear with the Cold War.

In fact, the world is currently facing many nuclear challenges:

• The US is modernising its nuclear arsenal.
• China is on track to quintuple their nuclear forces by 2030.
• Cooling relations between the US and Russia mean that existing arms reduction treaties (like New START) are looking very likely to lapse.
• Rising tensions in the Middle East, alongside the collapse of the Iran deal, mean we could very possibly see a new nuclear weapons state in the near future.
• And all that’s before we start talking about the war in Ukraine.

In this new nuclear age, we’re going to really need those nuclear experts.

So, we spoke to some existing experts to find out what you could be doing with your career to help. For example, you could be:

• Working in governments, especially the US Congress, Department of Defense, and Department of Energy
• Researching crucial questions, like how to keep a nuclear war from escalating after the first bomb has gone off
• Carrying out large-scale advocacy, like rebuilding public support for nuclear reductions

We think that working to reduce nuclear risk could be one of the best things you could do with your career, which is why we’ve added “nuclear weapons safety and security” to our list of the highest-impact career paths our research has identified so far.

So, why not take a moment to explore how your talents can play a role in preventing nuclear catastrophe and preserving peace for generations to come.

Learn more:

• Our problem profile on nuclear war
• Our blog post on why you might not want to work on nuclear disarmament (and what to work on instead)
• The new nuclear age — a collection of articles on US nuclear arsenal in Scientific American
• Our article on why we should try to protect future generations

The post Where are all the nuclear experts? appeared first on 80,000 Hours.

In other words, gain abilities that are valued in the job market — which makes your work more useful and makes it easier to bargain for the ingredients of a fulfilling job — as well as those that are specifically needed in tackling the world’s most pressing problems.

So today, we’re launching our series on the most useful skills for making a difference — which you can find here. It covers why we recommend each skill, how to get started learning them, and how to work out which is the best fit for you.

Each article looks at one of eight skill sets we think are most useful for solving the problems we think are most pressing:

• Policy and political skills

• Organisation-building

• Research

• Communicating ideas

• Software and tech skills

• Experience with an emerging power

• Engineering

• Expertise relevant to a top problem

Why are we releasing this now?

We think that many of our readers have come away from our site underappreciating the importance of career capital. Instead, they focus their career choices on having an impact right away.

This is a difficult tradeoff in general. Roughly, our position is that:

• There’s often less tradeoff between these things than people think, as good options for career capital often involve directly working on a problem you think is important.

• That said, building career capital substantially increases the impact you’re able to have. This is in part because the impact of different jobs is heavy-tailed, and career capital is one of the primary ways to end up in the tails.

• As a result, neglecting career capital can lower your long-term impact in return for only a small increase in short-term impact.

• Young people especially should be prioritising career capital in most cases.

We think that building career capital is important even for people focusing on particularly urgent problems — for example, we think that whether you should do an ML PhD doesn’t depend (much) on your AI timelines.

Why the focus on skills?

We break down career capital into five components:

• Skills and knowledge

• Connections

• Credentials

• Character

• Runway (i.e. savings)

We’ve found that “build useful skills” is a particularly good rule of thumb for building career capital.

It’s true that in addition to valuable skills, you also need to learn how to sell those skills to others and make connections. This can involve deliberately gaining credentials, such as by getting degrees or creating public demo projects; or it can involve what’s normally thought of as “networking,” such as going to conferences or building up a Twitter following. But all of these activities become much easier once you have something useful to offer.

The decision to focus on skills was also partly inspired by discussions with Holden Karnofsky and his post on building aptitudes, which we broadly agree with.

If you have more questions, take a look at our skills FAQ.

How can you help?

Please take a look at our new series and, if possible, share it with a friend!

We’d love feedback on these pages. If you have any, please do let us know by contacting us at info@80000hours.org.

Thank you so much!

The post Our new series on building skills appeared first on 80,000 Hours.

Bohlin came up with one very simple invention: the modern seat belt.

Within a year, Volvo began equipping their cars with seat belts as standard, and — as a result of its importance to safety — opened up the patent so that other manufacturers could do the same. Volvo claims that Bohlin’s invention has saved over a million lives. That would make Bohlin one of the highest-impact people in history, alongside David Nalin, the inventor of oral rehydration therapy for diarrhoea.

We’d guess Bohlin’s impact wasn’t quite that large. For one thing, seat belts already existed: in 1951, a Y-shaped three-point seat belt was patented that avoided the risks of internal injuries from simple lap belts. Bohlin’s innovation was doing this with just one strap, making it simple and convenient to use. For another thing, it seems likely that someone else would have come up with Bohlin’s design eventually.

Nevertheless, a simple estimate suggests that Bohlin saved hundreds of lives at the very least² — incredible for such a simple piece of engineering.

Thanks to Jessica Wen and Sean Lawrence at High Impact Engineers for their help with this article. Much of the content is based on their website.

Why are engineering skills valuable?

Bohlin’s story shows that engineering — by which we mean all kinds of engineering other than software engineering, which we cover separately — can clearly be hugely valuable for the world. But we think it’s most valuable when:

• You can really speed up development. This might be because you’re working on something that’s relatively neglected by others, or because you’re working in an area where you have high personal fit, so you can make particularly helpful contributions (or, ideally, both).
• You’re producing something which will practically be used to help people. One reason Bohlin had such a large impact — and Griswold, the inventor of the Y-shaped three-point seat belt didn’t — is that Volvo opened up the patent for use by other manufacturers.
• You’re working on a particularly pressing problem. For example, vaccines for common and deadly diseases — like malaria — are much more useful for the world than vaccines for rare diseases.

Ultimately, many of the potential solutions to the top problems we recommend working on include developing and deploying technology — and this often requires engineers.

Below, we look more closely at how engineering could be used to solve some of the world’s most pressing problems.

Jobs in engineering are often highly paid and in-demand. So learning engineering skills can give you great back-up options, and — depending on the specific discipline — can be a decent choice for earning to give.

Good pay combined with intellectually rewarding work means that engineers often have high job satisfaction (although we’d expect job satisfaction to be lower in academia than in industry).

Finally, it’s worth noting that it’s possible to accidentally cause harm through engineering. While we’re generally hugely in favour of technological development, many of the risks we’re most concerned about arise directly from the development of future technologies. Many technologies are dual use and could have both positive and negative applications. So it’s worth thinking carefully about the work you’re doing and whether it could be used to cause harm. (For an example of how you might think about this, see this article on whether it’s good to work on advancing AI capabilities. This example primarily applies to software engineers, but could also apply more broadly — to computer hardware engineers for instance.)

What specific discipline of engineering is most valuable?

There are many different types of engineering. Typically, you’ll eventually specialise in one (often during an undergraduate degree).

There are ways of using any engineering discipline to have an impact.

That said, we’re most excited about:

• Biomedical engineering
• Chemical engineering
• Electrical and electronic engineering.

That’s because these areas are most relevant to some of our top problems, in particular preventing catastrophic pandemics and reducing the risk of an AI-related catastrophe.

Some engineering disciplines also pay much better than others. In particular, nuclear, aerospace, petroleum and computer hardware engineers are paid best (although we wouldn’t generally recommend becoming a petroleum engineer, as we’d worry it causes harm), while agricultural and civil engineers are paid least.

What does using an engineering skill set typically involve?

An engineering skill set usually involves developing technologies faster and deploying existing technologies in novel ways. (This is in contrast with research skills, which focus on finding answers to unanswered questions, although there’s a fair bit of overlap between the two.)

Engineers typically do one of the following:

• Work in academia
• Work in industry
• Work at small startups (or found them)

Work in academia

Work in academia tends to focus on more speculative, early-stage technology (e.g. using ultraviolet light to sterilise rooms). This work is much more similar to research, so if you’re interested we’d suggest looking at our articles on research skills and working in academia. This route almost always involves getting a PhD in a subfield of engineering.

Academic research can be difficult for many people. It often involves long deadlines, self-driven work, and very little structure. Beyond engineering, academic work is also likely to include grant applications, teaching courses, publishing papers, mentoring students, and other responsibilities.

(We’ll look more at what to consider when choosing to do a PhD below.)

Work in industry or startups

As the technology becomes more viable, businesses tend to get involved — either startups or large engineering firms, or both. There are also some nonprofits focused on high-impact technology.

When working on engineering in industry, you can choose to become a subject matter expert (more similar to research) or instead become a manager, increasing the scope of your responsibilities. Either way, you can try learning faster by getting temporary placements in other parts of a company, taking part in engineering competitions, or working towards professional registration (which can be a helpful credential for engineering careers).

Generally, the work you focus on will be dictated by the business needs of the company, and, compared to academia, you’re more likely to have a standard 9-5 workday (rather than more flexible hours). Deadlines are often much shorter than in academia.

If you choose to become a manager or work for a small startup, you’ll be using organisation-building skills alongside your engineering skill set.

How to evaluate your fit

How to predict your fit in advance

You’ll need a quantitative background, and ideally you’ll have studied (or plan to study) engineering or a highly related subject at undergraduate level.

If you’re considering doing an engineering degree (or otherwise moving your career into engineering), signs you’d be a great fit could include:

• You’re comfortable working on scientific subjects.
• You’re good at practical, hands-on work: in many areas of engineering, you’ll end up working with physical objects in a lab.
• You enjoy understanding how and why physical things work.
• You enjoy fixing or building things, for example, around your home.
• You are good at “systems thinking”: for example, you’d notice when people ask you similar questions multiple times and then think about how to prevent the issue from coming up again.
• You might also be good at learning quickly and have high attention to detail.

With academic engineering, you’ll need to be comfortable with the academic research environment and generally happy to be self-motivated while working on things with few clear deadlines. If you’re doing a degree, you could try doing some sort of academic research (like a summer research project) and think about how that goes. (Read more about evaluating your fit for research.)

If you want to become a manager or work for a startup, you’ll probably need more social skills (including things like clear communication and people management skills).

Assessing your fit for different disciplines of engineering

One way to start is to think about which of the natural sciences you most enjoy learning about. Some examples:

Another way to determine what kind of engineering you might be good at is to figure out where you lie on the spectrum from scientist to engineer. If you enjoy the more theoretical, abstract, or precise side of physics or mathematics, then something like materials science or electrical engineering could be a better fit. If you lean more towards optimisation, application of knowledge, or practicalities, then civil or chemical engineering might be more interesting. If you are somewhere in the middle, then mechanical engineering could be for you.

However, don’t place too much weight on these crude tests — all these areas involve design testing and innovation, as well as research and studying new phenomena.

Your discipline also may not matter that much when it comes to getting a job. For example, many larger companies will hire graduate engineers from a range of different disciplines for the same role, relying on on-the-job training for specialisation.

How to tell if you’re on track

Within industry, the stages here look like an organisation-building career, and you can also assess your fit by looking at your rate of progression through the organisation.

Within academia, there’s generally very defined progression (e.g. completing a PhD, getting a postdoc, etc.).

In both cases, it’s worth trying to find some engineers whose work you respect, and who you trust to be honest with you, to give you feedback on how you’re getting along.

How to get started building engineering skills

Getting an engineering degree

The main way to get started is to do an undergraduate degree in engineering — although if you have a different quantitative degree, you may well be able to get an engineering job. (Read our advice on how to spend your time while at college.)

Engineering degrees are usually in a particular discipline of engineering. However, it can often be fairly easy to switch between engineering courses at university if you find that you’re not enjoying one kind of engineering.

Some universities may offer a ‘general first year’ for engineering in which you can take classes from different engineering disciplines to get a feel for what you enjoy.

Universities may have a range of student clubs or teams that work together to design, fabricate, test, and operate a complex vehicle or device in a national or worldwide competition with other universities. Examples include Formula SAE, the University Rover Challenge, UAS challenge, rocketry competitions (e.g. Australian Universities Rocket Competition), and human-powered vehicle challenges.

These sorts of competitions teach important skills that are invaluable in an engineering career — but they do typically require a large time commitment. Employers often view participation in these sorts of student teams very favourably, so it can give you a leg up in getting a job after graduating.

If you can, do internships in industry. Most large engineering companies run summer internships, and they are a good opportunity to see how industry works and gain some career capital. You could also do an engineering research project over the summer with a research group or join a startup. If all else fails, using the summer to create something also gives you valuable skills and experience — plus it lets you test out how much you like it.

Going into academia

If you want to do engineering in academia, you probably need to do a PhD.

Many people find PhDs very difficult. They can be isolating and frustrating, and take a very long time (4–6 years). What’s more, both your quality of life and the amount you’ll learn will depend on your supervisor — and it can be really difficult to figure out in advance whether you’re making a good choice.

So, if you’re considering doing a PhD, here are some things to consider:

• The topic of your research: It’s easy to let yourself be tied down to a PhD topic you’re not confident in. If the PhD you’re considering would let you work on something that seems relevant to a pressing problem you want to work on, it’s probably — all else equal — better for your career, and the research itself might have a positive impact as well.
• Mentorship: What are the supervisors or managers like at the opportunities open to you? You might be able to find engineering roles in industry where you could learn much more than you would in a PhD — or vice versa. When picking a supervisor, try reaching out to the current or former students of a prospective supervisor to ask them some frank questions. You can also use your final year undergraduate research project to evaluate your fit with a supervisor. (Also, see this article on how to choose a PhD supervisor.)
  Your fit for the work environment: Doing a PhD could mean working on your own with very little supervision or feedback for long periods of time. Some people thrive in these conditions! But some people really don’t and find PhDs extremely difficult.

PhD competitiveness varies by field. To get into any PhD, you’ll probably need high undergraduate grades and some research experience — including a reference from one or more professors. More competitive PhDs might require you to have published papers or extremely strong references. To get those, you might need to spend 1–3 years as a research assistant before applying for PhDs.

Entering industry

You can likely use an undergraduate degree to get an entry-level position in anything ranging from large engineering companies to startups.

In some countries (like the UK), large engineering companies offer graduate programs where you do rotations in different teams in the company. These allow you to build up lots of different skills and knowledge quickly (your ability to choose your rotation depends on the company, the department, and your manager).

Large companies are also likely to have a structured professional development scheme with training, assigned mentors, and regular check-ins to set you up for professional registration as an engineer.

Joining a startup generally means that you have a lot of responsibilities very quickly and less structure around you. This might mean more freedom with what you can do and lots of variety. You might learn a ton, but you won’t get much feedback or mentorship, and there will also be more stress and uncertainty.

Find jobs that use engineering

If you think you might be a good fit for this skill and you’re ready to start looking at job opportunities that are currently accepting applications, see our curated list of opportunities:

View all opportunities

Once you have an engineering skill set, how can you best apply it to have an impact?

Having a big impact as an engineer means finding a particularly pressing global issue and finding a way to use engineering to develop solutions.

Below is a list of pressing global problems and how engineers can help with each.

If you’re already an engineer, you can read through to see if any of these issues appeal to you — and then aim to speak to some people in each area about how your skills could be applied and what the current opportunities are.

You could also apply to speak to our team or get in touch with High Impact Engineers.

Preventing catastrophic pandemics

A future pandemic that is much worse than COVID-19 could pose a significant risk to society.

There’s a key role for bioengineers and chemical engineers to play in mitigating these risks, including:

• Developing vaccine platform technologies to help us rapidly produce new vaccines in response to novel threats
• Developing and implementing metagenomic sequencing to improve our ability to detect new pandemics

Other engineering disciplines are also needed. For example, engineers could:

• Help design better pathogen containment systems for labs and systems to reduce pathogen spread in buildings or vehicles. (There are roles here for materials, civil, industrial, aerospace, and HVAC engineers, among others.)
• Help improve stockpiling and management of PPE (personal protective equipment), such as gloves and masks. (There are possibly roles here for industrial engineers.)
• Help improve technologies for monitoring pathogens, like systems for sampling environments and processes for managing and examining samples. (There are roles here for industrial, mechanical, and automation engineers, among others.)

To learn more, take a look at Biosecurity needs engineers by Will Bradshaw and this overview of using engineering in biosecurity from High Impact Engineers.

AI alignment

We expect AI hardware to be a crucial component of the development of AI. Given the importance of positively shaping the development of AI, experts in AI hardware could be in a position to have a substantive positive impact.

Useful disciplines include:

• Electrical, electronic, and computer engineering (probably the most relevant discipline for AI hardware)
• Materials engineering with a focus on semiconductors
• Industrial engineering with a focus on the semiconductor supply chain

To learn more, read our full career review on becoming an expert in AI hardware.

If you have hardware expertise, you might also consider moving into AI policy. Read our career review of AI governance and coordination to learn more.

Improving civilisational resilience

One very neglected potential way to reduce existential threats is through generally increasing the resilience of our society to catastrophes.

All kinds of engineers can play a big role in this issue — for example by developing alternative foods, refuges, and knowledge stores that will be able to survive a near-apocalypse.

For instance, David Denkenberger is an engineer developing alternative foods that could be rapidly scaled up in the event of a global famine, perhaps caused by nuclear winter or a major volcanic eruption. We have two podcasts with him:

• Using paper mills and seaweed to feed everyone in a catastrophe
• We could feed all eight billion people through a nuclear winter

To learn more about refuges, see this review by Open Philanthropy. Or learn about how to increase the chance of recovery from a catastrophic event in two of our podcast episodes:

• Luisa Rodriguez on why global catastrophes seem unlikely to kill us all
• Paul Christiano on whether we should leave messages for future civilisations

Fight climate change

We think further developing and rolling out green energy is one of the best ways to tackle climate change, and engineers have a major role to play in this. This includes not just generating more green electricity, but also things like ensuring that there is enough electricity to meet seasonal changes in electricity demand and trying to find ways to make other forms of energy greener (like replacing fossil fuel use in blast furnaces or transportation).

You can further increase your impact by focusing on technology that’s either not widely known (e.g. hot rock geothermal) or unsexy (e.g. decarbonising cement rather than developing electric cars).

We have more notes on how to most effectively tackle climate change. We’d also recommend What can a technologist do about climate change? by Bret Victor.

Other problem areas that need engineers

In addition to the top problems mentioned above, there are many other pressing areas where engineers are needed. For example, you could:

• Create alternative proteins through chemical engineering to help reduce meat consumption and prevent the suffering of animals in factory farms.
• Work on tech and logistics for global health, like producing and rolling out vaccines for polio and malaria.
• Try to develop products for people living in poverty — see more about why that can be impactful in our interview with SendWave.

Options outside engineering that can use engineering aptitude

Engineers often have a systems mindset that can make them a particularly good fit for operations management or entrepreneurship. If that work interests you, it’s worth considering whether to spend some time building the skills you’d need to make the transition.

Some engineers may also excel at other options that require good quantitative abilities, such as:

• Working on AI safety technical research
• Software engineering
• Earning to give in a high-paying role, such as quantitative trading

Engineers may be able to easily develop skills in translating technically complex topics to less technical audiences, such as policymakers, which means you could also consider building a policy skill set. For example, TechCongress aims to get engineers, and other technologists, involved as technical advisors for policymakers.

Career paths we’ve reviewed that use engineering skills

• Expert in AI hardware
• Impactful climate solutions
• Biomedical research

Learn more about engineering

• High Impact Engineers
• Will Bradshaw on why biosecurity needs engineers and materials scientists
• A list of potential projects in biosecurity
• Podcast: Ken Goldberg on why your robot butler isn’t here yet

Read next:  Explore other useful skills

Want to learn more about the most useful skills for solving global problems, according to our research? See our list.

Continue →

The post Engineering skills appeared first on 80,000 Hours.

In a nutshell:

You can start building software and tech skills by trying out learning to code, and then doing some programming projects before applying for jobs. You can apply (as well as continue to develop) your software and tech skills by specialising in a related area, such as technical AI safety research, software engineering, or information security. You can also earn to give, and this in-demand skill set has great backup options.

Why are software and tech skills valuable?

By “software and tech” skills we basically mean what your grandma would call “being good at computers.”

When investigating the world’s most pressing problems, we’ve found that in many cases there are software-related bottlenecks.

For example, machine learning (ML) engineering is a core skill needed to contribute to AI safety technical research. Experts in information security are crucial to reducing the risks of engineered pandemics, as well as other risks. And software engineers are often needed by nonprofits, whether they’re working on reducing poverty or mitigating the risks of climate change.

Also, having skills in this area means you’ll likely be highly paid, offering excellent options to earn to give.

Moreover, basic programming skills can be extremely useful whatever you end up doing. You’ll find ways to automate tasks or analyse data throughout your career.

What does a career using software and tech skills involve?

A career using these skills typically involves three steps:

1. Learn to code with a university course or self-study and then find positions where you can get great mentorship. (Read more about how to get started.)
2. Optionally, specialise in a particular area, for example, by building skills in machine learning or information security.
3. Apply your skills to helping solve a pressing global problem. (Read more about how to have an impact with software and tech.)

There’s no general answer about when to switch from a focus on learning to a focus on impact. Once you have some basic programming skills, you should look for positions that both further improve your skills and have an impact, and then decide based on which specific opportunities seem best at the time.

Software and tech skills can also be helpful in other, less directly-related career paths, like being an expert in AI hardware (for which you’ll also need a specialist knowledge skill set) or founding a tech startup (for which you’ll also need an organisation-building skill set). Being good with computers is also often part of the skills required for quantitative trading.

Programming also tends to come in handy in a wide variety of situations and jobs; there will be other great career paths that will use these skills that we haven’t written about.

How to evaluate your fit

How to predict your fit in advance

Some indications you’ll be a great fit include:

• The ability to break down problems into logical parts and generate and test hypotheses
• Willingness to try out many different solutions
• High attention to detail
• Broadly good quantitative skills

The best way to gauge your fit is just to try out programming.

It seems likely that some software engineers are significantly better than average — and we’d guess this is also true for other technical roles using software. In particular, these very best software engineers are often people who spend huge amounts of time practicing. This means that if you enjoy coding enough to want to do it both as a job and in your spare time, you are likely to be a good fit.

How to tell if you’re on track

If you’re at university or in a bootcamp, it’s especially easy to tell if you’re on track. Good signs are that you’re succeeding at your assigned projects or getting good marks. An especially good sign is that you’re progressing faster than many of your peers.

In general, a great indicator of your success is that the people you work with most closely are enthusiastic about you and your work, especially if those people are themselves impressive!

If you’re building these skills at an organisation, signs you’re on track might include:

• You get job offers at organisations you’d like to work for.
• You’re promoted within your first two years.
• You receive excellent performance reviews.
• You’re asked to take on progressively more responsibility over time.
• After some time, you’re becoming someone in your team who people look to solve their problems, and people want you to teach them how to do things.
• You’re building things that others are able to use successfully without your input.
• Your manager / colleagues suggest you might take on more senior roles in the future.
• You ask your superiors for their honest assessment of your fit and they are positive (e.g. they tell you you’re in the top 10% of people they can imagine doing your role).

How to get started building software and tech skills

Independently learning to code

As a complete beginner, you can write a Python program in less than 20 minutes that reminds you to take a break every two hours.

A great way to learn the very basics is by working through a free beginner course like Automate the Boring Stuff with Python by Al Seigart.

Once you know the fundamentals, you could try taking an intro to computer science or intro to programming course. If you’re not at university, there are plenty of courses online, such as:

• Udacity’s Intro to Theoretical Computer Science
• MIT’s Introduction to Computer Science and Programming
• Stanford’s Programming Methodology

Don’t be discouraged if your code doesn’t work the first time — that’s what normally happens when people code!

A great next step is to try out doing a project with other people. This lets you test out writing programs in a team and working with larger codebases. It’s easy to come up with programming projects to do with friends — you can see some examples here.

Once you have some more experience, contributing to open-source projects in particular lets you work with very large existing codebases.

Attending a coding bootcamp

We’ve advised many people who managed to get junior software engineer jobs in less than a year by going to a bootcamp.

Coding bootcamps are focused on taking people with little knowledge of programming to as highly paid a job as possible within a couple of months. This is a great entry route if you don’t already have much background, though some claim the long-term prospects are not as good as if you studied at university or in a particularly thorough way independently because you lack a deep understanding of computer science. Course Report is a great guide to choosing a bootcamp. Be careful to avoid low-quality bootcamps. To find out more, read our interview with an App Academy instructor.

Studying at university

Studying computer science at university (or another subject involving lots of programming) is a great option because it allows you to learn to code in an especially structured way and while the opportunity cost of your time is lower.

It will also give you a better theoretical understanding of computing than a bootcamp (which can be useful for getting the most highly-paid and intellectually interesting jobs), a good network, some prestige, and a better understanding of lower-level languages like C. Having a computer science degree also makes it easier to get a US work visa if you’re not from the US.

Doing internships

If you can find internships, ideally at the sorts of organisations you might want to work for to build your skills (like big tech companies or startups), you’ll gain practical experience and the key skills you wouldn’t otherwise pick up from academic degrees (e.g. using version control systems and powerful text editors). Take a look at our our list of companies with software and machine learning internships.

AI-assisted coding

As you’re getting started, it’s probably worth thinking about how developments in AI are going to affect programming in the future — and getting used to AI-assisted coding.

We’d recommend trying out using GitHub CoPilot, which writes code for you based on your comments. Cursor is a popular AI-assisted code editor based on VSCode.

You can also just ask AI chat assistants for help. ChatGPT is particularly helpful (although only if you use the paid version).

We think it’s reasonably likely that many software and tech jobs in the future will be heavily based on using tools like these.

Building a specialty

Depending on how you’re going to use software and tech skills, it may be useful to build up your skills in a particular area. Here’s how to get started in a few relevant areas:

Depending on how you’re aiming to have an impact with these skills (see the next section), you may also need to develop other skills. We’ve written about some other relevant skill sets:

• How to get started building research skills
• How to get started building specialist knowledge relevant to a pressing problem
• How to get started building skills for communicating ideas

For more, see our full list of impactful skills.

Once you have these skills, how can you best apply them to have an impact?

The problem you work on is probably the biggest driver of your impact. The first step is to make an initial assessment of which problems you think are most pressing (even if you change your mind over time, you’ll need to decide where to start working).

Once you’ve done that, the next step is to identify the highest-potential ways to use software and tech skills to help solve your top problems.

There are five broad categories here:

• Use software and tech skills in research. Lots of technical research relevant to the world’s most pressing problems makes heavy use of software and tech skills — most notably, AI safety technical research. To be successful, you might also need a research skill set, which we’ve written about separately. For some paths, you’ll also need specialist knowledge in an area related to a pressing problem — e.g. hardware for becoming an expert in AI hardware.
• ML engineering for AI safety research. Most AI safety researchers work closely with engineers (and in many organisations, no clear distinction is made). This is a particularly high-impact way of using software and tech skills because we think risks from AI is one of the world’s most pressing problems.
• Build software for organisations working on pressing problems. Most organisations working on everything from global health to reducing the risk of nuclear war need software engineers to manage computer systems, apps, and websites. The key feature that draws this work together is that you’ll be building a product for others to use. Read more about software engineering careers and organisation-building skills.
• Protect sensitive information. Some organisations need help protecting information that could be hugely dangerous if it was known more widely, such as harmful genetic sequences or powerful AI technology. Breaches in areas like these could have disastrous consequences — which makes information security a great option for people who want to have a high-impact career. Read more about information security.
• Earn to give. Most jobs that use software and tech skills, whether software engineering, information security, data science, or something else entirely, command high salaries (particularly in the US) — and so they offer a great option for earning to give. Skilled software engineers can earn $300,000 a year or more at big tech companies. Probably the highest-paying routes are trading in quantitative hedge funds or founding a tech startup.

While some of these options (like protecting dangerous information) will require building up some more specialised skills, being a great programmer will let you move around most of these categories relatively easily, and the earning to give options means you’ll always have a pretty good backup plan.

Find jobs that use software and tech skills

See our curated list of job opportunities for this path.

View all opportunities

Career paths we’ve reviewed that use these skills

• AI safety technical research
• Information security in high-impact areas
• Expert in AI hardware
• Forecasting and related research and implementation
• Data collection for AI alignment
• Software engineering
• Found a tech startup
• Earning to give in quantitative trading
• Machine learning PhDs
• Early-stage startup employee
• Working at a leading AI lab
• Computer science PhD
• Data science
• Product manager in tech
• Web designer

Read next:  Explore other useful skills

Want to learn more about the most useful skills for solving global problems, according to our research? See our list.

Continue →

The post Software and tech skills appeared first on 80,000 Hours.

Many highly specific areas of knowledge seem applicable to solving the world’s most pressing problems, especially risks posed by biotechnology and artificial intelligence.

In particular we’d highlight:

• Subfields of biology relevant to pandemic prevention. Working on many of the possible technical solutions to reduce the risk of pandemics will require expertise in parts of biology. We’d particularly highlight synthetic biology, mathematical biology, virology, immunology, pharmacology, and vaccinology. This expertise can also be helpful for pursuing a biorisk-focused policy career. (Read more about careers to prevent catastrophic pandemics.)
• AI hardware. Specialised hardware is a crucial input to the development of frontier AI systems. As a result, we expect expertise in AI hardware to become increasingly important to the governance of AI systems. (Read more about becoming an expert in AI hardware).
• Economics. Understanding economics can be valuable in a huge range of impactful roles when combined with another skill set. For example, economics research is crucial for conducting global priorities research and improving decision making in large institutions. And a knowledge of economics can also support you in building policy and political skills, particularly for policy design and governance research.
• Other areas we sometimes recommend include history, knowledge of China, and law.

Of course, whatever skill set you focus on, you’ll likely need to build some specialist knowledge — for example, if you focus on policy and political skills, you’ll need to gain specialist knowledge in the area of policy you’re working in. Similarly, if you build software and tech skills, you could consider gaining specialist knowledge in machine learning or information security. The idea of the above list is just to highlight areas we think seem particularly valuable that you might not otherwise consider learning about.

How should you get started building specialist knowledge?

Each area is very different, so it’s hard to give any specific advice that applies to all of them.

Besides the articles on specific areas linked above, we’d suggest checking out:

• Whether to go to grad school
• How to find a career that fits you best
• How to be more successful in any job, especially the sections on learning how to learn and how to master a field and make creative contributions

All our career reviews relevant to building specialist knowledge

• Biorisk research, strategy, and policy
• Expert in AI hardware
• Investigate a potentially pressing but unexplored global issue
• Data collection for AI alignment
• Engineering
• Economics PhDs
• Law school in the US

Read next:  Explore other useful skills

Want to learn more about the most useful skills for solving global problems, according to our research? See our list.

Continue →

The post Specialist knowledge relevant to a top problem appeared first on 80,000 Hours.

It might not sound like much, but as a result of Borlaug’s research, wheat production in India and Pakistan almost doubled between 1965 and 1970, and formerly famine-stricken countries across the world were suddenly able to produce enough food for their entire populations. These developments have been credited with saving up to a billion people from famine,¹ and in 1970, Borlaug was awarded the Nobel Peace Prize.

Many of the highest-impact people in history, whether well-known or completely obscure, have been researchers.

Why are research skills valuable?

Not everyone can be a Norman Borlaug, and not every discovery gets adopted. Nevertheless, we think research can often be one of the most valuable skill sets to build — if you’re a good fit.

We’ll argue that:

• Research seems to have been extremely high-impact historically
• There are good theoretical reasons to think that research will be high-impact
• Research skills seem extremely useful to the problems we think are most pressing
• If you’re a good fit, you can have much more impact than the average
• And, depending on which subject you focus on, you may have good backup options.

Together, this suggests that research skills could be particularly useful for having an impact.

Later, we’ll look at:

• How to evaluate your fit for building research skills
• How to get started building research skills
• How you can use these skills to have an impact once you’ve started

Research seems to have been extremely high-impact historically

If we think about what has most improved the modern world, much can be traced back to research: advances in medicine such as the development of vaccines against infectious diseases, developments in physics and chemistry that led to steam power and the industrial revolution, and the invention of the modern computer, an idea which was first proposed by Alan Turing in his seminal 1936 paper On Computable Numbers.²

Many of these ideas were discovered by a relatively small number of researchers — but they changed all of society. This suggests that these researchers may have had particularly large individual impacts.

That said, research today is probably lower-impact than in the past. Research is much less neglected than it used to be: there are nearly 25 times as many researchers today as there were in 1930.³ It also turns out that more and more effort is required to discover new ideas, so each additional researcher probably has less impact than those that came before.⁴

However, even today, a relatively small fraction of people are engaged in research. As an approximation, only 0.1% of the population are academics,⁵ and only about 2.5% of GDP is spent on research and development. If a small number of people account for a large fraction of progress, then on average each person’s efforts are significant.

Moreover, we still think there’s a good case to be made for research being impactful on average today, which we cover in the next two sections.

There are good theoretical reasons to think that research will be high-impact

There’s little commercial incentive to focus on the most socially valuable research. And most researchers don’t get rich, even if their discoveries are extremely valuable. Alan Turing made no money from the discovery of the computer, and today it’s a multibillion-dollar industry. This is because the benefits of research often come a long time in the future and can’t usually be protected by patents. This means if you care more about social impact than profit, then it’s a good opportunity to have an edge.

Research is also a route to leverage. When new ideas are discovered, they can be spread incredibly cheaply, so it’s a way that a single person can change a field. And innovations are cumulative — once an idea has been discovered, it’s added to our stock of knowledge and, in the ideal case, becomes available to everyone. Even ideas that become outdated often speed up the important future discoveries that supersede it.

Research skills seem extremely useful to the problems we think are most pressing

When you look at our list of the world’s most pressing problems — like preventing future pandemics or reducing risks from AI systems — expert researchers seem like a key bottleneck.

For example, to reduce the risk posed by engineered pandemics, we need people who are talented at research to identify the biggest biosecurity risks and to develop better vaccines and treatments.

To ensure that developments in AI are implemented safely and for the benefit of humanity, we need technical experts thinking hard about how to design machine learning systems safely and policy researchers to think about how governments and other institutions should respond. (See this list of relevant research questions.)

And to decide which global priorities we should spend our limited resources on, we need economists, mathematicians, and philosophers to do global priorities research. For example, see the research agenda of the Global Priorities Institute at Oxford.

We’re not sure why so many of the most promising ways to make progress on the problems we think are most pressing involve research, but it may well be due to the reasons in the section above — research offers huge opportunities for leverage, so if you take a hits-based approach to finding the best solutions to social problems, it’ll often be most attractive.

In addition, our focus on neglected problems often means we focus on smaller and less developed areas, and it’s often unclear what the best solutions are in these areas. This means that research is required to figure this out.

For more examples, and to get a sense of what you might be able to work on in different fields, see this list of potentially high-impact research questions, organised by discipline.

If you’re a good fit, you can have much more impact than the average

The sections above give reasons why research can be expected to be impactful in general. But as we’ll show below, the productivity of individual researchers probably varies a great deal (and more than in most other careers). This means that if you have reason to think your degree of fit is better than average, your expected impact could be much higher than the average.

Depending on which subject you focus on, you may have good backup options

Pursuing research helps you develop deep expertise on a topic, problem-solving, and writing skills. These can be useful in many other career paths. For example:

• Many research areas can lead to opportunities in policymaking, since relevant technical expertise is valued in some of these positions. You might also have opportunities to advise policymakers and the public as an expert.
• The expertise and credibility you can develop by focusing on research (especially in academia) can put you in a good position to switch your focus to communicating important ideas, especially those related to your speciality, either to the general public, policymakers, or your students.
• If you specialise in an applied quantitative subject, it can open up certain high-paying jobs, such as quantitative trading or data science, which offer good opportunities for earning to give.

Some research areas will have much better backup options than others — lots of jobs value applied quantitative skills, so if your research is quantitative you may be able to transition into work in effective nonprofits or government. A history academic, by contrast, has many fewer clear backup options outside of academia.

What does building research skills typically involve?

By ‘research skills’ we broadly mean the ability to make progress solving difficult intellectual problems.

We find it especially useful to roughly divide research skills into three forms:

Academic research

Building academic research skills is the most predefined route. The focus is on answering relatively fundamental questions which are considered valuable by a specific academic discipline. This can be impactful either through generally advancing a field of research that’s valuable to society or finding opportunities to work on socially important questions within that field.

Turing was an academic. He didn’t just invent the computer — during World War II he developed code-breaking machines that allowed the Allies to be far more effective against Nazi U-boats. Some historians estimate this enabled D-Day to happen a year earlier than it would have otherwise.⁶ Since World War II resulted in 10 million deaths per year, Turing may have saved about 10 million lives.

We’re particularly excited about academic research in subfields of machine learning relevant to reducing risks from AI, subfields of biology relevant to preventing catastrophic pandemics, and economics — we discuss which fields you should enter below.

Academic careers are also excellent for developing credibility, leading to many of the backup options we looked at above, especially options in communicating important ideas or policymaking.

Academia is relatively unique in how flexibly you can use your time. This can be a big advantage — you really get time to think deeply and carefully about things — but can be a hindrance, depending on your work style.

See more about what academia involves in our career review on academia.

Practical but big picture research

Academia rewards a focus on questions that can be decisively answered with the methods of the field. However, the most important questions can rarely be answered rigorously — the best we can do is look at many weak forms of evidence and come to a reasonable overall judgement. which means while some of this research happens in academia, it can be hard to do that.

Instead, this kind of research is often done in nonprofit research institutes, e.g. the Centre for the Governance of AI or Our World in Data, or independently.

Your focus should be on answering the questions that seem most important (given your view of which global problems most matter) through whatever means are most effective.

Some examples of questions in this category that we’re especially interested in include:

• How likely is a pandemic worse than COVID-19 in the next 10 years?
• How difficult is the AI alignment problem going to be to solve?
• Which global problems are most pressing?
• Is the world getting better or worse over time?
• What can we learn from the history of philanthropy about which forms of philanthropy might be most effective?

You can see a longer list of ideas in this article.

Someone we know who’s had a big impact with research skills is Ajeya Cotra. Ajeya initially studied electrical engineering and computer science at UC Berkeley. In 2016, she joined Open Philanthropy as a grantmaker.⁷ Since then she’s worked on a framework for estimating when transformative AI might be developed, how worldview diversification could be applied to allocating philanthropic budgets, and how we might accidentally teach AI models to deceive us.

Applied research

Then there’s applied research. This is often done within companies or nonprofits, like think tanks (although again, there’s also plenty of applied research happening in academia). Here the focus is on solving a more immediate practical problem (and if pursued by a company, where it might be possible to make profit from the solution) — and there’s lots of overlap with engineering skills. For example:

• Developing new vaccines
• Creating new types of solar cells or nuclear reactors
• Developing meat substitutes

Neel was doing an undergraduate degree in maths when he decided that he wanted to work in AI safety. Our team was able to introduce Neel to researchers in the field and helped him secure internships in academic and industry research groups. Neel didn’t feel like he was a great fit for academia — he hates writing papers — so he applied to roles in commercial AI research labs. He’s now a research engineer at DeepMind. He works on mechanistic interpretability research which he thinks could be used in the future to help identify potentially dangerous AI systems before they can cause harm.

We also see “policy research” — which aims to develop better ideas for public policy — as a form of applied research.

Stages of progression through building and using research skills

These different forms of research blur into each other, and it’s often possible to switch between them during a career. In particular, it’s common to begin in academic research and then switch to more applied research later.

However, while the skill sets contain a common core, someone who can excel in intellectual academic research might not be well-suited to big picture practical or applied research.

The typical stages in an academic career involve the following steps:

1. Pick a field. This should be heavily based on personal fit (where you expect to be most successful and enjoy your work the most), though it’s also useful to think about which fields offer the best opportunities to help tackle the problems you think are most pressing, give you expertise that’s especially useful given these problems, and use that at least as a tie-breaker. (Read more about choosing a field.)
2. Earn a PhD.
3. Learn your craft and establish your career — find somewhere you can get great mentorship and publish a lot of impressive papers. This usually means finding a postdoc with a good group and then temporary academic positions.
4. Secure tenure.
5. Focus on the research you think is most socially valuable (or otherwise move your focus towards communicating ideas or policy).

Academia is usually seen as the most prestigious path…within academia. But non-academic positions can be just as impactful — and often more so since you can avoid some of the dysfunctions and distractions of academia, such as racing to get publications.

At any point after your PhD (and sometimes with only a master’s), it’s usually possible to switch to applied research in industry, policy, nonprofits, and so on, though typically you’ll still focus on getting mentorship and learning for at least a couple of years. And you may also need to take some steps to establish your career enough to turn your attention to topics that seem more impactful.

Note that from within academia, the incentives to continue with academia are strong, so people often continue longer than they should!

If you’re focused on practical big picture research, then there’s less of an established pathway, and a PhD isn’t required.

Besides academia, you could attempt to build these skills in any job that involves making difficult, messy intellectual judgement calls, such as investigative journalism, certain forms of consulting, buy-side research in finance, think tanks, or any form of forecasting.

Personal fit is perhaps more important for research than other skills

The most talented researchers seem to differ hugely in their impact compared to typical researchers across a wide variety of metrics and according to the opinions of other researchers.

For instance, when we surveyed biomedical researchers, they said that very good researchers were rare, and they’d be willing to turn down large amounts of money if they could get a good researcher for their lab.⁸ Professor John Todd, who works on medical genetics at Cambridge, told us:

The best people are the biggest struggle. The funding isn’t a problem. It’s getting really special people[…] One good person can cover the ground of five, and I’m not exaggerating.

This makes sense if you think the distribution of research output is very wide — that the very best researchers have a much greater output than the average researcher.

How much do researchers differ in productivity?

It’s hard to know exactly how spread out the distribution is, but there are several strands of evidence that suggest the variability is very high.

Firstly, most academic papers get very few citations, while a few get hundreds or even thousands. An analysis of citation counts in science journals found that ~47% of papers had never been cited, more than 80% had been cited 10 times or less, but the top 0.1% had been cited more than 1,000 times. A similar pattern seems to hold across individual researchers, meaning that only a few dominate — at least in terms of the recognition their papers receive.

Citation count is a highly imperfect measure of research quality, so these figures shouldn’t be taken at face-value. For instance, which papers get cited the most may depend at least partly on random factors, academic fashions, and “winner takes all” effects — papers that get noticed early end up being cited by everyone to back up a certain claim, even if they don’t actually represent the research that most advanced the field.

However, there are other reasons to think the distribution of output is highly skewed.

William Shockley, who won the Nobel Prize for the invention of the transistor, gathered statistics on all the research employees in national labs, university departments, and other research units, and found that productivity (as measured by total number of publications, rate of publication, and number of patents) was highly skewed, following a log-normal distribution.

Shockley suggests that researcher output is the product of several (normally distributed) random variables — such as the ability to think of a good question to ask, figure out how to tackle the question, recognize when a worthwhile result has been found, write adequately, respond well to feedback, and so on. This would explain the skewed distribution: if research output depends on eight different factors and their contribution is multiplicative, then a person who is 50% above average in each of the eight areas will in expectation be 26 times more productive than average.⁹

When we looked at up-to-date data on how productivity differs across many different areas, we found very similar results. The bottom line is that research seems to perhaps be the area where we have the best evidence for output being heavy-tailed.

Interestingly, while there’s a huge spread in productivity, the most productive academic researchers are rarely paid 10 times more than the median, since they’re on fixed university pay-scales. This means that the most productive researchers yield a large “excess” value to their field. For instance, if a productive researcher adds 10 times more value to the field than average, but is paid the same as average, they will be producing at least nine times as much net benefit to society. This suggests that top researchers are underpaid relative to their contribution, discouraging them from pursuing research and making research skills undersupplied compared to what would be ideal.

Can you predict these differences in advance?

Practically, the important question isn’t how big the spread is, but whether you could — early on in your career — identify whether or not you’ll be among the very best researchers.

There’s good news here! At least in scientific research, these differences also seem to be at least somewhat predictable ahead of time, which means the people entering research with the best fit could have many times more expected impact.

In a study, two IMF economists looked at maths professors’ scores in the International Mathematical Olympiad — a prestigious maths competition for high school students. They concluded that each additional point scored on the International Mathematics Olympiad “is associated with a 2.6 percent increase in mathematics publications and a 4.5 percent increase in mathematics citations.”

We looked at a range of data on how predictable productivity differences are in various areas and found that they’re much more predictable in research.

What does this mean for building research skills?

The large spread in productivity makes building strong research skills a lot more promising if you’re a better fit than average. And if you’re a great fit, research can easily become your best option.

And while these differences in output are not fully predictable at the start of a career, the spread is so large that it’s likely still possible to predict differences in productivity with some reliability.

This also means you should mainly be evaluating your long-term expected impact in terms of your chances of having a really big success.

That said, don’t rule yourself out too early. Firstly, many people systematically underestimate their skills. (Though others overestimate them!) Also, the impact of research can be so large that it’s often worth trying it out, even if you don’t expect you’ll succeed. This is especially true because the early steps of a research career often give you good career capital for many other paths.

How to evaluate your fit

How to predict your fit in advance

It’s hard to predict success in advance, so we encourage an empirical approach: see if you can try it out and look at your track record.

You probably have some track record in research: many of our readers have some experience in academia from doing a degree, whether or not they intended to go into academic research. Standard academic success can also point towards being a good fit (though is nowhere near sufficient!):

• Did you get top grades at undergraduate level (a 1st in the UK or a GPA over 3.5 in the US)?
• If you do a graduate degree, what’s your class rank (if you can find that out)? If you do a PhD, did you manage to author an article in a top journal (although note that this is easier in some disciplines than others)?

Ultimately, though, your academic track record isn’t going to tell you anywhere near as much as actually trying out research. So it’s worth looking for ways to cheaply try out research (which can be easy if you’re at college). For example, try doing a summer research project and see how it goes.

Some of the key traits that suggest you might be a good fit for a research skills seem to be:

• Intelligence (Read more about whether intelligence is important for research.)
• The potential to become obsessed with a topic (Becoming an expert in anything can take decades of focused practice, so you need to be able to stick with it.)
• Relatedly, high levels of grit, self-motivation, and — especially for independent big picture research, but also for research in academia — the ability to learn and work productively without a traditional manager or many externally imposed deadlines
• Openness to new ideas and intellectual curiosity
• Good research taste, i.e. noticing when a research question matters a lot for solving a pressing problem

There are a number of other cheap ways you might try to test your fit.

Something you can do at any stage is practice research and research-based writing. One way to get started is to try learning by writing.

You could also try:

• Finding out what the prerequisites/normal backgrounds of people who go into a research area are to compare your skills and experience to them
• Reading key research in your area, trying to contribute to discussions with other researchers (e.g. via a blog or twitter), and getting feedback on your ideas
• Talking to successful researchers in a field and asking what they look for in new researchers

How to tell if you’re on track

Here are some broad milestones you could aim for while becoming a researcher:

• You’re successfully devoting time to building your research skills and communicating your findings to others. (This can often be the hardest milestone to hit for many — it can be hard to simply sustain motivation and productivity given how self-directed research often needs to be.)
• In your own judgement, you feel you have made and explained multiple novel, valid, nontrivially important (though not necessarily earth-shattering) points about important topics in your area.
• You’ve had enough feedback (comments, formal reviews, personal communication) to feel that at least several other people (whose judgement you respect and who have put serious time into thinking about your area) agree, and (as a result) feel they’ve learned something from your work. For example, lots of this feedback could come from an academic supervisor. Make sure you’re asking people in a way that gives them affordance to say you’re not doing well.
• You’re making meaningful connections with others interested in your area — connections that seem likely to lead to further funding and/or job opportunities. This could be from the organisations most devoted to your topics of interest; but, there could also be a “dissident” dynamic in which these organisations seem uninterested and/or defensive, but others are noticing this and offering help.

If you’re finding it hard to make progress in a research environment, it’s very possible that this is the result of that particular environment, rather than the research itself. So it can be worth testing out multiple different research jobs before deciding this skill set isn’t for you.

Within academic research

Academia has clearly defined stages, so you can see how you’re performing at each of these.

Very roughly, you can try asking “How quickly and impressively is my career advancing, by the standards of my institution and field?” (Be careful to consider the field as a whole, rather than just your immediate peers, who might be very different from average.) Academics with more experience than you may be able to help give you a clear idea of how things are going.

We go through this in detail in our review of academic research careers.

Within independent research

As a very rough guideline, people who are an excellent fit for independent research can often reach the broad milestones above with a year of full-time effort purely focusing on building a research skill set, or 2–3 years of 20%-time independent effort (i.e. one day per week).

Within research in industry or policy

The stages here can look more like an organisation-building career, and you can also assess your fit by looking at your rate of progression through the organisation.

How to get started building research skills

As we mentioned above, if you’ve done an undergraduate degree, one obvious pathway into research is to go to graduate school (read our advice on choosing a graduate programme) and then attempt to enter academia before deciding whether to continue or pursue positions outside of academia later in your career.

If you take the academic path, then the next steps are relatively clear. You’ll want to try to get excellent grades in undergraduate and in your master’s, ideally gain some kind of research experience in your summers, and then enter the best PhD programme you can. From there, focus on learning your craft by working under the best researcher you can find as a mentor and working in a top hub for your field. Try to publish as many papers as possible since that’s required to land an academic position.

It’s also not necessary to go to graduate school to become a great researcher (though this depends a lot on the field), especially if you’re very talented.
For instance, we interviewed Chris Olah, who is working on AI research without even an undergraduate degree.

You can enter many non-academic research jobs without a background in academia. So one starting point for building up research skills would be getting a job at an organisation specifically focused on the type of question you’re interested in. For examples, take a look at our list of recommended organisations, many of which conduct non-academic research in areas relevant to pressing problems.

More generally, you can learn research skills in any job that heavily features making difficult intellectual judgement calls and bets, preferably on topics that are related to the questions you’re interested in researching. These might include jobs in finance, political analysis, or even nonprofits.

Another common route — depending on your field — is to develop software and tech skills and then apply them at research organisations. For instance, here’s a guide to how to transition from software engineering into AI safety research.

If you’re interested in doing practical big-picture research (especially outside academia), it’s also possible to establish your career through self-study and independent work — during your free time or on scholarships designed for this (such as EA Long-Term Future Fund grants and Open Philanthropy support for individuals working on relevant topics).

Some example approaches you might take to self-study:

• Closely and critically review some pieces of writing and argumentation on relevant topics. Explain the parts you agree with as clearly as you can and/or explain one or more of your key disagreements.
• Pick a relevant question and write up your current view and reasoning on it. Alternatively, write up your current view and reasoning on some sub-question that comes up as you’re thinking about it.
• Then get feedback, ideally from professional researchers or those who use similar kinds of research in their jobs.

It could also be beneficial to start with some easier versions of this sort of exercise, such as:

• Explaining or critiquing interesting arguments made on any topic you find motivating to write about
• Writing fact posts
• Reviewing the academic literature on any topic of interest and trying to reach and explain a bottom-line conclusion

In general, it’s not necessary to obsess over being “original” or having some new insight at the beginning. You can learn a lot just by trying to write up your current understanding.

Choosing a research field

When you’re getting started building research skills, there are three factors to consider in choosing a field:

1. Personal fit — what are your chances of being a top researcher in the area? Even if you work on an important question, you won’t make much difference if you’re not particularly good at it or motivated to work on the problem.
2. Impact — how likely is it that research in your field will contribute to solving pressing problems?
3. Back-up options — how will the skills you build open up other options if you decide to change fields (or leave research altogether)?

One way to go about making a decision is to roughly narrow down fields by relevance and back-up options and then pick among your shortlist based on personal fit.

We’ve found that, especially when they’re getting started building research skills, people sometimes think too narrowly about what they can be good at and enjoy. Instead, they end up pigeonholing themselves in a specific area (for example being restricted by the field of their undergraduate degree). This can be harmful because it means people who could contribute to highly important research don’t even consider it. This increases the importance of writing a broad list of possible areas to research.

Given our list of the world’s most pressing problems, we think some of the most promising fields to do research within are as follows:

• Fields relevant to artificial intelligence, especially machine learning, but also computer science more broadly. This is mainly to work on AI safety directly, though there are also many opportunities to apply machine learning to other problems (as well as many back-up options).
• Biology, particularly synthetic biology, virology, public health, and epidemiology. This is mainly for biosecurity.
• Economics. This is for global priorities research, development economics, or policy research relevant to any cause area, especially global catastrophic risks.
• Engineering — read about developing and using engineering skills to have an impact.
• International relations/political science, including security studies and public policy — these enable you to do research into policy approaches to mitigating catastrophic risks and are also a good route into careers in government and policy more broadly.
• Mathematics, including applied maths or statistics (or even physics). This may be a good choice if you’re very uncertain, as it teaches you skills that can be applied to a whole range of different problems — and lets you move into most of the other fields we list. It’s relatively easy to move from a mathematical PhD into machine learning, economics, biology, or political science, and there are opportunities to apply quantitative methods to a wide range of other fields. They also offer good back-up options outside of research.
• There are many important topics in philosophy and history, but these fields are unusually hard to advance within, and don’t have as good back-up options. (We do know lots of people with philosophy PhDs who have gone on to do other great, non-philosophy work!)

However, many different kinds of research skills can play a role in tackling pressing global problems.

Choosing a sub-field can sometimes be almost as important as choosing a field. For example, in some sciences the particular lab you join will determine your research agenda — and this can shape your entire career.

And as we’ve covered, personal fit is especially important in research. This can mean it’s easily worth going into a field that seems less relevant on average if you are an excellent fit. (This is due both to the value of the research you might produce and the excellent career capital that comes from becoming top of an academic field.)

For instance, while we most often recommend the fields above, we’d be excited to see some of our readers go into history, psychology, neuroscience, and a whole number of other fields. And if you have a different view of global priorities from us, there might be many other highly relevant fields.

Once you have these skills, how can you best apply them to have an impact?

Richard Hamming used to annoy his colleagues by asking them “What’s the most important question in your field?”, and then after they’d explained, following up with “And why aren’t you working on it?”

You don’t always need to work on the very most important question in your field, but Hamming has a point. Researchers often drift into a narrow speciality and can get detached from the questions that really matter.

Now let’s suppose you’ve chosen a field, learned your craft, and are established enough that you have some freedom about where to focus. Which research questions should you focus on?

Which research topics are the highest-impact?

Charles Darwin travelled the oceans to carefully document different species of birds on a small collection of islands — documentation which later became fuel for the theory of evolution. This illustrates how hard it is to predict which research will be most impactful.

What’s more, we can’t know what we’re going to discover until we’ve discovered it, so research has an inherent degree of unpredictability. There’s certainly an argument for curiosity-driven research without a clear agenda.

That said, we think it’s also possible to increase your chances of working on something relevant, and the best approach is to try to find topics that both personally motivate you and seem more likely than average to matter. Here are some approaches to doing that.

Using the problem framework

One approach is to ask yourself which global problems you think are most pressing, and then try to identify research questions that are:

• Important to making progress on those problems (i.e. if this question were answered, it would lead to more progress on these problems)
• Neglected by other researchers (e.g. because they’re at the intersection of two fields, unpopular for bad reasons, or new)
• Tractable (i.e. you can see a path to making progress)

The best research questions will score at least moderately well on all parts of this framework. Building a perpetual motion machine is extremely important — if we could do it, then we’d solve our energy problems — but we have good reason to think it’s impossible, so it’s not worth working on. Similarly, a problem can be important but already have the attention of many extremely talented researchers, meaning your extra efforts won’t go very far.

Finding these questions, however, is difficult. Often, the only way to identify a particularly promising research question is to be an expert in that field! That’s because (when researchers are doing their jobs), they will be taking the most obvious opportunities already.

However, the incentives within research rarely perfectly line up with the questions that most matter (especially if you have unusual values, like more concern for future generations or animals). This means that some questions often get unfairly neglected. If you’re someone who does care a lot about positive impact and have some slack, you can have a greater-than-average impact by looking for them.

Below are some more ways of finding those questions (which you can use in addition to directly applying the framework above).

Rules of thumb for finding unfairly neglected questions

• There’s little money in answering the question. This can be because the problem mostly affects poorer people, people who are in the future, or non-humans, or because it involves public goods. This means there’s little incentive for businesses to do research on this question.
• The political incentives to answer the question are missing. This can happen when the problem hurts poorer or otherwise marginalised people, people who tend not to organise politically, people in countries outside the one where the research is most likely to get done, people who are in the future, or non-humans. This means there’s no incentive for governments or other public actors to research this question.
• It’s new, doesn’t already have an established discipline, or is at the intersection of two disciplines. The first researchers in an area tend to take any low hanging fruit, and it gets harder and harder from there to make big discoveries. For example, the rate of progress within machine learning is far higher than the rate of progress within theoretical physics. At the same time, the structure of academia means most researchers stay stuck within the field they start in, and it can be hard to get funding to branch out into other areas. This means that new fields or questions at the intersection of two disciplines often get unfairly neglected and therefore provide opportunities for outsized impact.
• There is some aspect of human irrationality that means people don’t correctly prioritise the issue. For instance, some issues are easy to visualise, which makes them more motivating to work on. People are scope blind which means they’re likely to neglect the issues with the very biggest scale. They’re also bad at reasoning about issues with low probability, which can make them either over-invest or under-invest in them.
• Working on the question is low status. In academia, research that’s intellectually interesting and fits the research standards of the discipline are high status. Also, mathematical and theoretical work tends to be seen as higher status (and therefore helps to progress your career). But these don’t correlate that well with the social value of the question.
• You’re bringing new skills or a new perspective to an established area. Progress often comes in science from bringing the techniques and insights of one field into another. For instance, Kahneman started a revolution in economics by applying findings from psychology. Cross-over is an obvious approach but is rarely used because researchers tend to be immersed in their own particular subject.

If you think you’ve found a research question that’s short on talent, it’s worth checking whether the question is answerable. People might be avoiding the question because it’s just extremely difficult to find an answer. Or perhaps progress isn’t possible at all. Ask yourself, “If there were progress on this question, how would we know?”

Finally, as we’ve discussed, personal fit is particularly important in research. So position yourself to work on questions where you maximise your chances of producing top work.

Find jobs that use a research skills

If you have these skills already or are developing it and you’re ready to start looking at job opportunities that are currently accepting applications, see our curated list of opportunities for this skill set:

View all opportunities

Career paths we’ve reviewed that use these skills

• AI safety technical research and engineering
• AI governance and coordination
• Biorisk research
• China-related AI safety and governance paths
• Grantmaker focused on pressing world problems
• Research into global priorities
• Forecasting and related research and implementation
• Historian of large societal trends, inflection points, progress or collapse
• Expert in AI hardware

• Specialist in emerging global powers

• Investigate a potentially pressing but unexplored global issue
• Research management
• Think tank research
• Academic research
• Research and advocacy promoting impactful climate solutions
• Improving China-Western coordination on global catastrophic risks
• Engineering
• Economics PhDs
• Machine learning PhDs
• Biomedical research
• Computer science PhDs
• Data science
• Philosophy academia

Learn more about research

• High Impact Science by Carl Shulman
• How to succeed as an early-stage researcher: the “lean startup” approach
• Podcast: Luisa and Robert Long on how to make independent research more fun
• A list of potentially high-impact research questions, organised by discipline

See all our articles and podcasts on research careers.

Read next:  Explore other useful skills

Want to learn more about the most useful skills for solving global problems, according to our research? See our list.

Continue →

The post Research skills appeared first on 80,000 Hours.

This logic doesn’t just apply to criminal justice. For almost any global issue you’re interested in, roles in powerful institutions like governments often offer unique and high-leverage ways to address some of the most pressing challenges of our time.

Why are policy and political skills valuable?

We’ll argue that:

• Governments (and other powerful institutions) have a huge impact in the world in general.
• Powerful institutions are often important for finding solutions to the problems we think are most pressing.
• You can create change if you’re able to build the skills that will let you do so.
• You’ll need to use your influence well and avoid causing harm.

Together, this suggests that building the skills needed to get things done in large institutions could give you a lot of opportunities to have an impact.

Later, we’ll look at:

• How to evaluate your fit for building policy and political skills
• How to get started building policy and political skills
• How you can use these skills to have an impact once you’ve started

Governments (and other powerful institutions) have a huge impact in the world

National governments are hugely powerful.

For a start, they command the spending of huge sums of money. The US government’s federal budget is approximately $6.4 trillion/year — that’s approximately the annual revenue of the world’s 14 largest companies by revenue (although only around $1.7 trillion/year is discretionary spending). Many other Western countries spend hundreds of billions of dollars a year.

And it’s not just money. Governments produce laws governing the actions of millions — or billions — and have unique tools at their disposal, including taxation and tax breaks, regulation, antitrust actions, and, ultimately, the use of force.

The US spends nearly a trillion dollars a year on its military (although this is an outlier — in other Western countries it’s more like tens of billions).

Why does this scale matter?

Well, we’ll argue that your chances of reaching a government role in which you can have a large influence are probably high enough that in expectation you can have a significant impact, given the huge scale of government action.

And it’s not just governments. Most of the advice in this article can be applied to any powerful institution, such as an international body or organisation like the United Nations. Much of what we say even applies to jobs at large corporations.

Governments and other major institutions play a major role in addressing the world’s most pressing problems

National governments and international bodies — in particular the US, UK, and EU — are already working on some of the problems we have identified as most pressing. For example:

• Biorisk: The UK government released the UK Biological Security Strategy aimed at preventing future pandemics in June 2023. The US Centers for Disease Control and Prevention (CDC) works on public health in the US and is also one of the most important organisations working on global disease control. The US defence and intelligence community also works in this area. For instance, the Department of Defense does a lot of work on infectious diseases and assists other countries’ efforts to prevent the proliferation of biological weapons.
• AI safety and public policy: In her annual State of the Union Address, the President of the European Commission told the European Parliament that the EU should be working to mitigate the risk of extinction from AI. The White House Office issued an executive order on AI, which — among other things — requires developers of the most powerful AI systems to develop safety standards and tests and share these results with the US government. The Defense Advanced Research Projects Agency (DARPA) has a program on explainable AI, which is a component of AI safety research. The UK government has set up the AI Safety Institute. And as AI becomes more important, governments will likely become more involved.
• Nuclear security: The US has the world’s most powerful military and the second biggest stockpile of nuclear weapons. Federal agencies such as the Department of Defense, the Department of Energy, and the State Department are important for preventing nuclear catastrophe.

Governments also play a major role in pretty much every other global issue you can think of (including basically every issue we have have profiles on, such as global health, climate change, and factory farming).

Throughout this article, we focus on the US because we think it has particular influence in areas related to the problems we think are most pressing, and because it’s where we have the most readers. However, we think these skills are also valuable to build if you’re based in many other countries (and we also have advice specifically about the UK).

Beyond governments, there are also international organisations and large companies that are important for solving certain problems. For example, the Biological Weapons Convention plays a unique role in preventing biological catastrophes, while leading AI labs and large tech companies have a crucial influence over the development of AI.

To see lists of particularly relevant institutions for various problems, see our problem profiles and job board.

You can create change

You might think that, even if you work at an important institution, you won’t have much impact because you won’t really be able to affect anything. You’ll have to carry out the will of elected officials, who are bound to the electorate, institutional constraints, and special interests. And while this is definitely true in many cases, we do think there are opportunities to have at least a small effect on the actions of these large and powerful institutions.

Frances Kelsey was an academic and a pharmacologist. But, in 1960, she took a major career step when she was hired by the FDA. Just one month into her new career in government, she was given her first assignment to review a drug: thalidomide. Despite considerable pressure from the drug’s manufacturer, Kelsey insisted that it be tested more rigorously.

And so, while more than 10,000 children across the world were born with birth defects as a result of thalidomide — living with life-long deformed limbs and defective organs — only 17 such children were born in the US. Kelsey was hailed by the American public as a hero and was awarded the President’s Award for Distinguished Federal Civilian Service in 1962.¹

But why was a mid-level official — only one month into her new job — able to have such an impact?

First, there’s just a huge amount to do, and senior officials don’t have that much time.

For example, in the US, there are 535 members of Congress and around 4,000 presidential appointees in the executive branch. That might sound like a lot, but think about it this way: each of these people, on average, has oversight over about 0.02% of the US federal budget — over $1 billion. It would be literally impossible to micromanage that amount of activity.

This is only a very rough heuristic, but by dividing the $1.7 trillion discretionary federal budget by the number of people at different levels of seniority, we can estimate the average budget that different subsets of people in the government oversee.²

Note that this method is just an estimate of the average and there are some reasons to think it’s probably too high.³

Nevertheless, these figures are so high that if you can help those budgets be used just a little more efficiently, it could be worth millions of dollars of additional spending in the area of focus.

And, in other ways, this is an underestimate of the responsibility of each individual because much of what the government does is not best thought of as setting budgets — rather it comes from regulation, foreign policy, changing social norms and so on. Budgets here are just being used as a proxy for one form of impact.

Second, the views and opinions of others in government aren’t completely fixed. Otherwise — whether you think it’s protected free speech or a distortion of democracy — it’s hard to explain why private companies spend around $4 billion a year on federal lobbying. For every dollar spent by a profit-oriented company on lobbying, it’s probably getting more than a dollar back on average by affecting government policy. This suggests that people interested in social change can have an impact, especially if they’re focused on global issues with little other lobbying, or they can find neglected ways to affect policy.

And so it’s not surprising that when we’ve spoken to people working in and around governments, we’ve found that — as in the case of Frances Kelsey — people have actually had the opportunity to influence things even in junior roles (if they had the skills).

In the US, we spoke to a number of mid-level and senior federal employees, and most were able to give us an example of how they had a large positive impact through their role. Some of their examples involved starting new impactful programs worth $10s of millions, saving Americans $100s of millions, or moving billions to something potentially more impactful. We haven’t vetted these stories, but at the very least they persuaded us that mid-level and senior federal employees feel as though they can sometimes have a large positive influence on the government.

In the UK, one junior civil servant we spoke to determined how £250 million was spent in her policy area through careful discussions with senior civil servants, while ministers were only scrutinising larger chunks of money.

And it’s not just in the executive. For example, in the US Congress, huge amounts of work are done by congressional staffers. “Ninety-five percent of the nitty-gritty work of drafting bills and negotiating their final form is now done by staff,” according to former Senator Ted Kennedy.⁴

Often this work is done by very junior people. One junior staff member in a Congressional office told us that more senior individuals (like Chiefs of Staff) are often tasked with substantial managerial responsibilities that crowd out their ability to focus on nitty-gritty policy research. Because of this, they have to defer to more junior staff (such as legislative assistants) who have the capacity and time to dig into a specific policy area and make concrete proposals.

This all suggests that you can effect change in large institutions (even when you’re just getting started), and in particular:

• On issues where people care enough for changes to be made, but not enough to micromanage the changes
• Where powerful figures like elected officials have vague goals, but no specific idea of what they want
• When details have a large impact, e.g. the details of one piece of legislation can affect many other laws

All other things being equal, the more senior you are, the more influence you’ll have.

If you’re a motivated graduate from a top university, over the course of your career, the chance of reaching high levels in the government is significant.

Approximately 1 in 30 federal employees in DC are in the senior executive service. What’s more, we found that students with a strong academic background and great social skills (and an interest in politics) in the UK could have an around 1 in 3 chance of becoming an MP. Meanwhile, if you became a Congressional staffer in the US, you’d have something like a 1 in 40 chance of being elected to Congress.

Other factors will also affect your ability to create change, such as how politicised your area is (the more political, the more your moves will be countered by others).

All that said, many people we speak to in the civil service don’t feel that they have a lot of influence. That’s because many roles don’t have opportunities for a lot of impact. (We’ll discuss finding ones that do later, and it can be hard to see your impact even in those that do.)

But the potential for change is there. You can think of decision making in large institutions as a negotiation between different groups with power. Most of the time you won’t tip the balance, but occasionally you might be able to — and it could have a large impact.

But you’ll need to use your influence responsibly

Having influence is a double-edged sword.

If you use your position poorly, then you might make things worse than they would have been otherwise. This is especially easy in policy, because it’s hard to know what truly makes things better, and policy can have unintended consequences. This is especially disturbing if you end up working on critical problems, such as preventing pandemics or nuclear crises.

This doesn’t mean you should avoid these positions altogether. For a start, someone has to take these positions, and it’ll probably be better for the world if more altruistic people enter them. Hopefully, if you’re reading this article, you’re more likely than average to be one of these people.

However, it does mean that if you succeed in advancing you have a huge responsibility to use the position well — and the higher you advance, the more responsibility you have.

This means trying to do the best job you can to help the institution do more good for society, and being especially careful to avoid actions that could cause significant harm.

Unfortunately, the more you advance, the easier it is to lose touch with people who will give you frank feedback, and the more temptations you’ll face to do unethical or dishonest actions in order to preserve your influence or “for the greater good” — i.e. to get corrupted.

This means we’d especially encourage people considering this path to focus on building good character and making sure they have friends around them who can keep them honest at the early stages, so these are in place in case they gain a lot of influence.

It’s also important to make sure you have a clear ‘edge’ that will allow you to do more good than a typical employee. For instance, you might be able to give ministers more evidence-based advice, contribute specialist knowledge, or pay more attention to the effect of policies on the long-term future than typical.

That said, even talented and very well-meaning people can fail to do good in government and even do harm, so it is worth learning constantly and thinking carefully and critically about what will actually help. Read more advice on avoiding harm.

What does using a policy and political skill set involve?

Any career path that ends up in an influential institutional position could be a way of using these skills, though some options are more likely to be relevant to the problems we think are most pressing.

This typically involves the following steps:

1. Identify some institutions that could play an important role tackling some of the problems you think are most pressing. See an introduction to comparing global problems in terms of impact and lists of institutions that are important to each area in our problem profiles and job board.

2. Learn to make useful contributions to an institution (or group of institutions) by gaining experience, credibility, seniority, and authority.

3. Often, it involves developing a speciality that’s especially relevant to the problems you want to focus on. For instance, if you want to work on tackling engineered pandemics, you might specialise in counter-terrorism, technology policy, or biomedical policy. This is both to help you advance into more relevant roles, but also to improve your understanding of which policies are actually helpful. That said, many policy makers remain generalists. In that case, you need to make sure you find trusted expert advisors to help you understand which policy changes would be most helpful.

4. Move into roles that put you in a better position to help tackle these problems. Focusing on pandemics again, you might aim to work at the Center for Disease Control and Prevention and then advance to more senior positions.

5. Have an impact by using your position and expertise to improve policies and practices relevant to pressing global problems or bringing attention to neglected but important priorities.

Within this skill set, it’s possible to focus more on policy research or policy implementation. The first is about developing ideas for new policies, and involves an element of applied research skills, while the second is a bit more like an organisation building skill set and has an impact via making an important institution more efficient.

There’s also a spectrum of roles from roles that are more like being a technical specialist to those — like roles in political parties or running for elected office — that are more political and closer to engagement with the general public and current affairs.

In addition to roles actually within the relevant institutions, there are also “influencer” roles which aim to shape these institutions from the outside.

This includes jobs in think tanks, advocacy non-profits, journalism, academia, and even corporations, rather than within government.

The skills needed for influencer roles are similar to those needed for policy and political roles in many ways, but they also overlap a lot with skills in research and communicating ideas. These roles can be a better fit for someone who wants to work in a smaller organisation, is less comfortable with political culture, or wants to focus more on ideas rather than application.

In practice, people often move between influencer and government positions across their careers.

Some people think that to work in policy you have to be brilliant at networking.

That’s not quite true — as we’ve seen, depending on your role, you might focus more on understanding and researching policies, communicating ideas to a specific audience, or just really understanding your particular institution very well.

But it’s nevertheless true that networking skills are more important in building a policy and political skill set than, for example, if you wanted to work in a purely research — and you can learn more about how to network in our article on how to be successful in any job. In particular, multiple people — both in the US and in the UK — have told us that it’s important to be friendly and nice to others.

Finally, we’d like to emphasise the potential value of doing policy-style work in industry, especially if you’re interested in AI policy. While government policy is likely to play a key role in coordinating various actors interested in reducing the risks from advanced AI, internal policy, compliance work, lobbying, and corporate governance within the largest AI labs are also powerful tools. Collaboration between labs and government also requires work that may use similar skills, like stakeholder management, policy design, and trust-building.

Example people

How to evaluate your fit

This skill set is fairly broad, which also means it can potentially be a good fit for a wide variety of people. Don’t rule it out based on a hazy sense that government work isn’t for you!

For example, entering policy through building specific expertise can be a good fit for people interested in research careers but who would like to do something more practical. Many roles are totally unlike the stereotype of a politician endlessly shaking hands or what ‘government bureaucrat’ brings to mind.

How to predict your fit in advance

Here are some traits that seem likely to point towards being a great fit:

• You have the potential to succeed at relationship-building and fitting in. In many of these roles, you need to be able to develop good relationships with a wide range of people in a short amount of time, come across as competent and warm in your interactions, genuinely want to add value and help others achieve their goals, consistently follow up and stay in touch with people, and build a reputation and be remembered.

  It helps to have empathy and social intelligence so that you can model other people’s viewpoints and needs accurately. It also helps if you can remember small details about people! You don’t necessarily need all these skills when you start out, but you should be interested in improving them.

  These skills are most important in more public-facing party-political positions and are also needed to work in large institutions. However, there are also roles focused more on applying technical expertise to policy, which don’t require these skills as much (though they’re still probably more important than in e.g. academia).

• You can think of a relevant institution at which you can imagine yourself being relatively happy, productive, and motivated for a long time — while playing by the institution’s rules. Try speaking with later-career people at the institution to get as detailed a sense as possible of how long it will take to reach the kind of position you’re hoping for, what your day-to-day life will be like in the meantime, and what you will need to do to succeed.

• Having the right citizenship. There are lots of influential and important policy roles in every country, so you should consider them wherever you live. But some roles in the US seem especially impactful — as do certain roles at large institutions like the EU. In particular, any of the roles within the US most relevant to the problems we think are most pressing — particularly in the executive branch and Congress — are only open to, or at least will heavily favour, American citizens. All key national security roles that might be especially important will be restricted to those with US citizenship, which is required to obtain a security clearance.

  If you’re excited about US policy in particular and are curious about immigration pathways and types of policy work available to non-citizens, see this blog post. Consider also participating in the annual diversity visa lottery if you’re from an eligible country, as this is low-effort and allows you to win a US green card if you’re lucky (getting a green card is the only way to become a citizen).

• Being comfortable with political culture. The culture in politics, especially US federal politics, can be difficult to navigate. Some people we know have entered promising policy positions, but later felt like the culture was a terrible fit for them. Experts we’ve spoken to say that, in Washington, DC, there’s a big cultural focus on networking and internal bureaucratic politics to navigate. We’ve also been told that while merit matters to a degree in US government work, it is not the primary determinant of who is most successful. We’d expect this to be similar in other countries. People who think they wouldn’t feel able or comfortable to be in this kind of environment for the long term should consider whether other skills or institutions would be a better fit.

  That said, this does vary substantially by area and by role. Some roles, like working in a parliament or somewhere like the White House, are much more exposed to politics than others. Also, if you work on a hot button, highly partisan issue, you’re much more likely to be exposed to intense political dynamics than if you work on more niche, technocratic, or cross-party issues.

It’s useful if you can find ways to do cheap tests first, like speaking to someone in the area (which could take a couple of hours), or doing an internship (which could take a couple of months). But often, you’ll need to take a job in the area to tell whether this is a good fit for you — and be willing to switch after a year or more if it’s not. For more, read our article on finding a job that fits you.

How to tell if you’re on track

First, ask yourself “How quickly and impressively is my career advancing, by the standards of the institution I’m currently focused on?” People with more experience (and advancement) at the institution will often be able to help you get a clear idea of how this is going. (It’s also just generally important to have good enough relationships with some experienced people to get honest input from them — this is an additional indicator of whether you’re “on track” in most situations.)

One caveat to this is that the rate of advancement could really vary depending on the exact role you have in that institution. For example, in Congress, speed of promotion often has to do less with your abilities and more with timing and the turnover of the office. As a result, the better the office, the fewer people leave and the slower the pace of promotion; the opposite is often true for bad offices. So you need to make sure you’re judging yourself by relevant standards — again, people with more experience at the institution should be able to help here.

Another relevant question to ask is “How sustainable does this feel?” This question is relevant for all skills, but especially here — for government and policy roles, one of the main things that affects how well you advance is simply how long you can stick with it and how consistently you meet the institution’s explicit and implicit expectations. So, if you find you can enjoy government and political work, that’s a big sign you’re on track. Just being able to thrive in government work can be an extremely valuable comparative advantage.

One other way to advance your career in government, especially as it relates to a specific area of policy, is what some call “getting visibility” — that is, using your position to learn about the landscape and connect with the actors and institutions that affect the policy area you care about. You’ll want to be invited to meetings with other officials and agencies, be asked for input on decisions, and engage socially with others who work in the policy area. If you can establish yourself as a well-regarded expert on an important but neglected aspect of the issue, you’ll have a better shot at being included in key discussions and events.

How to get started building policy and political skills

There are two main ways you might get started:

1. Institution-first. You’d start your career by trying to find a set of institutions that are a good fit for you and that seems at least relevant to the problems you think are most pressing (e.g. the executive branch of the US government or tech companies). You’d then try to move up the ranks of those institutions.
2. Expertise-first. In this route, you initially focus on building a relevant speciality or area of expertise (e.g. in academia or think tanks) and then use that to switch into institutional positions later. In addition, people with impressive credentials and accomplishments outside of government (e.g. in business, consulting, or law) can sometimes enter important departments and agencies at particularly senior and influential levels.

If you take the institution-first approach, you can try for essentially any job at this institution and focus on performing well by the institution’s standards. All else being equal, it’d be better to work on jobs relevant to a pressing problem, but just trying to advance should probably be your main goal early in your career.

The best way to learn how to perform and advance is to speak to people a couple of steps ahead of you in the path. Also look at cases of people who advanced unusually quickly and try to unpack what they did.

Sometimes the best way to advance will involve going somewhere other than the institution itself temporarily. For instance, going to law school, public policy school, or working at think tanks can give you credentials and connections that open up positions in government later.

If you’re focused on developing expertise in a particular area of policy, then it’s common to go to graduate school in a subject relevant to that area (e.g. economics, machine learning, biology).

As always, whether these paths are a good way of building your skills depends on the specific job or programme and people you’ll be working with:

• Will you get good mentorship?
• What’s their reputation in the field?
• Do they have good character?
• Does their policy agenda seem positive?
• Will the culture be a good fit for you?

With all that in mind, here are a few next steps that are especially good for building these skills:

Fellowships and leadership schemes

Fellowships can be an effective way to gain experience inside government or think tanks and can help you advance quickly into more senior government positions.

Some fellowships are aimed at people who already have some professional experience outside of policy but want to pivot into government roles, while others are aimed at recent graduates.

In the US, consider the Presidential Management Fellows for recent graduates of advanced degrees, the Horizon Fellowship, the AAAS fellowship for people with science PhDs or engineering master’s, or the TechCongress fellowship for mid-career tech professionals. If you have completed a STEM graduate degree, also consider the Mirzayan Science and Technology Policy Graduate Fellowship Program.

In the UK, try the Civil Service Fast Stream. And if you’re interested in EU AI policy, you can apply for the EU Tech Policy Fellowship. We also curate a list of UK / EU policy master’s options through our job board.

Graduate school

In general, we’d most recommend grad school for economics or machine learning. (Read more about why these are the best subjects to study at grad school.)

Some other useful subjects to highlight, given our list of pressing problems, include:

• Other applied quantitative subjects, like computer science, physics, and statistics
• Security studies, international relations, public policy, or law school, particularly for entering government and policy careers
• Subfields of biology relevant to pandemic prevention (like synthetic biology, mathematical biology, virology, immunology, pharmacology, or vaccinology)

Many master’s programmes offer specific coursework on public policy, science and society, security studies, international relations, and other topics. Having a graduate degree or law degree will give you a leg up for many positions.

In the US, a policy master’s, a law degree, or a PhD is particularly useful if you want to climb the federal bureaucracy. Choosing a graduate school near or close to DC is often a good idea, especially if you’re hoping to work part- or even full-time in public policy alongside graduate school.

While you’re studying (either at grad school or as an undergraduate), internships — for example in DC — are a promising route to evaluate your fit for policy work and to establish early career capital. Many academic institutions in the US offer a “Semester in DC” programme, which can let you explore placements of choice in Congress, federal agencies, or think tanks. The Virtual Student Federal Service (VSFS) also offers part-time, remote government internships.

Just bear in mind that graduate schools present the risk that you could spend a long time there without learning much about the actual career you’re pursuing itself or the problem you want to work on. It may sometimes make sense to try out a junior role or internship, see how it feels, and make sure you’re expecting a graduate degree to be worth it before going for it.

Read more about going to grad school.

Working for a politician or on a political campaign

Working for a politician as a researcher or staffer (e.g. as a parliamentary researcher in the UK, legislative staff for a Member of Congress, or as campaign staff for an electoral candidate) can be one useful step into political and policy positions. It’s also demanding, prestigious (especially in the US, less so in the UK), and gives you lots of connections. From this step, it’s also common to move into the executive branch or to later run for office. Read more in our career review on becoming a congressional staffer.

You don’t strictly need a master’s or other advanced degree to work in the US Congress. But many staffers still eventually pursue a graduate degree, in part because federal agencies and think tanks commonly care more about formal credentials, and many congressional staffers at some point switch to these institutions.

You can also work for a politician on a particular campaign — some of the top people who work on winning campaigns eventually get high-impact positions in the federal government. This is a high-risk strategy: it often only pays off if your candidate wins, and even then, not everybody on the campaign staff will get influential jobs or jobs in the areas they care about, especially if you’re a junior campaign staffer. (Running for office yourself involves a similar high-risk, high-reward dynamic.)

Roles in the executive branch

Look for entry-level roles in your national government, again focusing on positions at the executive-branch equivalent or those most relevant to policy-making.

In the US, you could take an entry-level role as a federal employee, ideally working on something relevant to a problem you want to help solve or will give you the flexibility to potentially work on multiple pressing problems. The most influential positions are usually in the executive branch.

That said, most people have told us that, in the US, it’s even better to get a graduate degree first because it will allow you to reach higher levels of career advancement and seniority more quickly. A graduate degree could also qualify you for fellowships.

In the UK, see our profile on civil service careers.

Think tank roles

Think tanks are organisations that aren’t part of government but still focus on informing and ultimately influencing policymaking.

Research roles at policy think tanks involve conducting in-depth research on specific policy areas and formulating relevant recommendations. These researchers also often collaborate with experts, host events, engage with policymakers, and liaise with the media to influence and inform public policy discourse. This often involves fundraising, grant writing, and staying updated on political trends — and it can teach you many of the skills that are useful in government.

These roles are relatively competitive and you may have your reputation tied to particular institutions you work for — which can have upsides and downsides.

Think tanks also employ non-research staff in communications, HR, finance, and other areas; these roles are less likely to meaningfully impact policy outcomes, though they could still be a reasonable way to build policy career capital.

Also, think tank staff are often fairly cleanly split between entry-level employees and senior employees with advanced degrees (often PhDs), with relatively few mid-level roles. For this reason, it’s fairly uncommon for people to stay and rise through the ranks at a think tank without leaving for graduate school or another role.

These roles let you learn about important policy issues and can open up many options in policy. One option is to continue working in think tanks or other influencer positions, perhaps specialising in an area of policy. Otherwise, it’s common to switch from think tanks to the executive branch, a campaign, or other policy positions.

(Read more in our career review on working in think tanks.)

Other options

It’s also common to enter policy and government jobs from consulting and law, as well as other professional services, public relations, and business in general.

More broadly, having organisation-building skills (e.g. public relations, organisational communications, finance, and accounting knowledge) or research skills can help you find policy and political roles.

Find jobs that use policy and political skills

If you think you might be a good fit for this skill set and you’re ready to start looking at job opportunities that are currently accepting applications, see our curated list of opportunities.

View all opportunities

Once you have these skills, how can you best apply them to have an impact?

Let’s suppose you now have a position with some ability to get things done in an important institution, and, from building expertise or an advisory network in particular pressing problems, you also have some ideas about the most important things you’d like to see happen. Then what should you do?

Depending on the issue and your position, you might then seek to have an impact via:

1. Improving the implementation of policy relevant to a pressing problem. For example, you could work at an agency regulating synthetic biology.

2. Gathering support for policy ideas. For example, you could highlight the top areas of consensus in the field about promising ways the government could reduce global poverty to a politician you work for.

3. Coming up with ideas for new policies. For example, you might craft new proposals for implementing compute governance policies.

Improving the implementation of policies

When people think about political careers, they usually think of people in suits having long debates about what to do.

But fundamentally, a policy is only an idea. For an idea to have an impact, someone actually has to carry it out.

The difference between the same policy carried out badly vs. competently can be enormous. For instance, during COVID-19, some governments reacted much faster than others, saving the lives of thousands of citizens.

What’s more, many policies are by necessity, only defined vaguely. For instance, a set of drug safety standards might need to show there is “reasonable evidence” a drug is safe, but — as shown by Frances Kelsey — how that is interpreted is left up to the relevant agency and may even change over time.

Many details are often left undecided when the policy is created, and again, these get filled out by government employees.

This option especially requires skills like people and project management, planning, coordination in and out of government, communication, resource allocation, training, and more.

So, if you can become great at one or more of these things (and really know your way around the institution you work in), it’s worth trying to identify large projects that might help solve the problems you think are most pressing — and then helping them run better.

These roles are most commonly found in the executive branch such as the Defense Department, the State Department, intelligence agencies, or the White House. (See also our profile on the UK civil service.)

Bringing ideas for new policies to the attention of important decision makers

One way to have an impact is to help get issues “on the agenda” by getting the attention and buy-in of important people.

For example, when politicians take office, they often enter on a platform of promises made to their constituents and their supporters about which policy agendas they want to pursue. They can be, to varying degrees, problem-specific — for example, having a broad remit of “improving health care.” Or, it could be more solution-specific — for example, aiming to create a single-payer health system or remove red tape facing critical industries. These agendas are formed through public discussion, media narratives, internal party politics, deliberative debate, interest group advocacy, and other forms of input. Using any of these ways to get something on the agenda is a great way to help make sure it happens.

You can contribute to this process in political advisory positions (e.g. being a staffer for a congressperson) or through influencer positions, such as think tanks.

As a rule of thumb, if you’re working within an institution (such as a large corporation or a government department), you want to be as senior as possible while still being responsible for a specific set of issues. In such a position, you’ll be in contact with all the key stakeholders, from the most senior people to those more on your level.

But it’s important to remember that, for many important issues, policymakers or officials at various levels of government can also prioritise solving certain problems or enacting specific proposals that aren’t the subject of national debate. In fact, sometimes making issues too salient, framing them in divisive ways, or allowing partisanship and political polarisation to shape the discussion, can make it harder to successfully get things done.

Coming up with ideas for new policies

In many areas relevant to particularly pressing problems, there’s a lack of concrete policies that are ready to implement.

Policy creation is a long process, often starting from broad intellectual ideas, which are iteratively developed into more practical proposals by think tanks, civil servants, political parties, advocates, and others, and then adjusted in response to their reception by peers, the media and the electorate, as well as political reality at the time.

Once concrete policy options are on the table, they must be put through the relevant decision-making process and negotiations. In countries with strong judicial review like the US, special attention often has to be paid to make sure laws and regulations will hold up under the scrutiny of the courts.

All this means there are many ways to contribute to policy creation in roles ranging from academia to government employees.

Many policy details are only hashed out at the later stages by civil servants and political advisors. This also means there isn’t a bright line between policy creation and policy implementation — more a spectrum that blurs from one into the other.

In the corporate context, internal policy creation can serve similar functions. Though they may be less enforceable unless backed up with contracts, the norms policies create can shape behaviour considerably.

While policy research is the bread and butter of think tank work, many staffers in Congress, agencies, and the White House also develop policy ideas or translate existing ideas into concrete policy proposals. For many areas of technical policy, especially AI policy, some of the best policy research is being done at industry labs, like OpenAI and DeepMind. (Read more about whether you should take a job at a top AI lab.)

For more details on the complex work of policy creation, we recommend Thomas Kalil’s article Policy Entrepreneurship in the White House: Getting Things Done in Large Organisations.

Career paths we’ve reviewed that use these skills

• AI governance and coordination
• Biorisk research, strategy, and policy
• Policy careers focused on other pressing global issues
• Research and advocacy promoting impactful climate solutions
• Congressional staffer
• Economics PhDs
• Law school in the US
• Working at a leading AI lab
• Party politics in the UK
• UK civil service jobs
• US policy master’s degrees

Learn more about government and policy

• Emergingtechpolicy.org, a website with expert advice and resources on US emerging tech policy (AI, bio, etc.)
• Working in US AI policy
• Podcast: Ezra Klein on existential risk from AI and what DC could do about it
• Podcast: Tom Kalil on how to have a big impact in government & huge organisations, based on 16 years’ experience in the White House
• Podcast: Tantum Collins on what he’s learned as an AI policy insider
• Collection of work on ‘Should you should focus on the EU if you’re interested in AI governance for longtermist/x-risk reasons?’ on the Effective Altruism Forum

See all our materials on policy and political careers.

Read next:  Explore other useful skills

Want to learn more about the most useful skills for solving global problems, according to our research? See our list.

Continue →

The post Policy and political skills appeared first on 80,000 Hours.

• Read the new and updated career guide here, by our founder Benjamin Todd and the 80,000 Hours team.

Our 2023 career guide isn’t just a fancy new design — here’s a rundown of how the content has been updated:

1. Career capital: get good at something useful

In our previous career guide, we argued that your primary focus should be on building very broadly applicable skills, credentials, and connections — what we called transferable career capital.

We also highlighted jobs like consulting as a way to get this.

However, since launching the 2017 version of the career guide, we came to think a focus on transferable career capital might lead you to neglect experience that can be very useful to enter the most impactful jobs — for example, experience working in an AI lab or studying synthetic biology.

OK, so how should you figure out the best career capital option for you?

Our new advice: get good at something useful.

In more depth — choose some valuable skills to learn, and that are a good fit for you, and then find opportunities that let you practise those skills. And then have concrete back-up plans and plan Bs in mind, rather than relying on general ‘transferability.’

This focus on skills is important because you’re much more likely to have an impact if you’re good at what you do — and research suggests it can take years of experience to reach your peak abilities. It also becomes much easier to build up other components of career capital — like gaining credentials or making connections — once you have something useful to offer.

We’ve supplemented this with an updated list of impactful role types to aim at long-term and common types of next steps that help to learn skills useful for those.

These steps still often involve learning skills that can be applied to many different global problems or sectors (since all else equal, more transferability is better), but we don’t emphasise transferability as much. We’re also less keen on consulting as a route into working on the most pressing problems (though it’s still best for some).

More:

• The chapter of our new guide on career capital
• Why you should consider other things before going into consulting

2. How to plan your career

We have greatly expanded our content on how to plan your career.

Our chapter on career planning leads you through planning for both a longer-term vision and immediate next steps:

• Your longer-term vision is useful for helping shape your plans, although it shouldn’t be more than a vague idea about where you’d like to end up (read more).
• You can then work backwards from that vision to help come up with next steps — but you should also work forward from your current situation, looking at any opportunities immediately in front of you (read more).

And to help you develop your career plan, we also have a new career planning template, designed to be used alongside our career guide.

More:

• The chapter of our new guide on career planning

3. Other changes and improvements

• New types of impactful careers. We added sections on why government and policy and organisation-building careers could have a high impact.
• A new chapter on which global problems are most pressing. The previous version of the printed book (although not the website) didn’t contain anything about which problems we think are most pressing and why. The new chapter tells the story of how our views have evolved, and why we focus on reducing existential risks today.
• Avoiding doing harm with your career. In the past years, we’ve become more concerned about the risk of people potentially causing harm with their careers, despite attempts to do good. Our new career guide more carefully and explicitly warns against this, and provides advice on how to avoid causing harm. Relatedly, we suggest considering your character as part of your career capital, and so considering how any job you take will shape and form your virtues.
• We greatly expanded the chapter on assessing personal fit and exploring your options.
• We’ve fully updated the more empirical sections of the guide using more up-to-date papers and data.

By working together, in our lifetimes, we can prevent the next pandemic and mitigate the risks of AI, we can end extreme global poverty and factory farming — and we can do this while having interesting, fulfilling lives too.

Our hope is that this new guide will help you do exactly that.

Learn more:

• Read the entire new career guide.
• Get the guide as a printed book (you can buy it on Amazon or email book.giveaway@80000hours.org from an address subscribed to our newsletter, and we’ll send you a free copy), on Audible, or for free as a podcast series and ebook (available as a .pdf or .epub).

The post What you should know about our updated career guide appeared first on 80,000 Hours.