Judgement, which I roughly define as ‘the ability to weigh complex information and reach calibrated conclusions,’ is clearly a valuable skill.
In our simple analysis of which skills make people most employable, using data from the Bureau of Labor Statistics across the US economy, ‘judgement and decision making’ came out top (though meant in a broader sense than we do).
My guess is that good judgement is even more important when aiming to have a positive impact.
What follows are some notes on why good judgement matters, what it is, and what we know about how to improve it.
Table of Contents
One reason is lack of feedback. We can never be fully certain which issues are most pressing, or which interventions are most effective. Even in an area like global health – where we have relatively good data on what works – there has been huge debate over the cost effectiveness of even a straightforward intervention like deworming. Deciding whether to focus on deworming requires judgement.
This lack of feedback becomes even more pressing when we come to efforts to reduce existential risks or help the long-term future, and efforts that take a more ‘hits based’ approach to impact. An existential risk can only happen once, so there’s a limit to how much data we can ever have about what reduces them, and we must mainly rely on judgement. 1
Reducing existential risks and some of the other areas we focus on are also new fields of research, so we don’t even have established heuristics or widely accepted knowledge that someone can simply learn and apply in place of using their judgement.
You may not need to make these judgement calls yourself – but you at least need to have good enough judgement to pick someone else with good judgement to listen to.
In contrast, in other domains it’s easier to avoid relying on judgement. For instance, in the world of for-profit startups, it’s possible (somewhat) to try things, gain feedback by seeing what creates revenue, and refine from there. Someone with so-so judgement can use other approaches to pursue a good strategy.
Other fields have other ways of avoiding judgement. In engineering you can use well-established quantitative rules to figure out what works. When you have lots of data, you can use statistical models. Even in more qualitative research like anthropology, there are standard ‘best practice’ research methods that people can use. In other areas you can follow traditions and norms that embody centuries of practical experience.
I get the impression that many in effective altruism agree that judgement is a key trait. In the 2020 EA Leaders Forum survey, respondents were asked which traits they would most like to see in new community members over the next five years, and judgement came out highest by a decent margin.
| Good judgement (weighing complex information and reaching calibrated conclusions) | 5.8 |
| Analytical intelligence | 5.1 |
| Entrepreneurial mindset (being able to make things happen independently) | 5.0 |
| Independent thinking (developing one's own views) | 5.0 |
| Altruism/Prioritizing the common good | 4.6 |
| Honesty/Transparency | 4.4 |
| Emotional intelligence/Social skills | 4.3 |
| Grit and work ethic | 3.6 |
| Ambition | 3.3 |
| Creativity | 3.1 |
It’s also notable that two of the other most desired traits – analytical intelligence and independent thinking – both relate to what we might call ‘good thinking’ as well. (Though note that this question was only about ‘traits,’ as opposed to skills/expertise or other characteristics.)
I think this makes sense. Someone with unusually good and trusted judgement can decide what an organisation’s strategy should be, or make large grants. This is valuable in general, though the community also seems to be especially constrained by people able to do this kind of work right now due to the funding overhang. Many of the bottlenecks faced by the community right now also involve research, and this requires a lot of judgement. When we’ve looked into the traits required to succeed in our priority paths, good judgement usually seems very important.
One promising feature of good judgement is that it seems more possible to improve than raw intelligence. So, what – more practically – is good judgement, and how can one get it?
I introduced a rough definition above, but there’s a lot of disagreement about what exactly good judgement is, so it’s worth saying a little more. Many common definitions seem overly broad, making judgement a central trait almost by definition. For instance, the Cambridge Dictionary defines it as:
The ability to form valuable opinions and make good decisions
While the US Bureau of Labor Statistics defines it as:
Considering the relative costs and benefits of potential actions to choose the most appropriate one
I prefer to focus on the rough narrower definition I introduced at the start (and which was used in the survey I mentioned above), which makes judgement more clearly different from other cognitive traits:
The ability to weigh complex information and reach calibrated conclusions
More practically, I think of someone with good judgement as someone able to:
Owen Cotton-Barratt wrote out his understanding of good judgement, breaking it into ‘understanding’ and ‘heuristics.’ His notion is a bit broader than mine.
Here are some closely related concepts:
Here are some other concepts in the area, but that seem more different:
How to improve judgement is an unsolved problem. The best overview I’ve found on what’s currently known is Open Philanthropy’s review of the research into methods to improve the judgement of their staff by Luke Muelhauser. The following suggestions are aligned with what they conclude.
In particular, the suggestions draw significantly on Phil Tetlock’s research into how to improve forecasting. This is the single best body of research I’m aware of in the area of improving judgement.
Tetlock’s research stands out from other research into improving decision making because:
Tetlock wrote a great popular summary of his research, Superforecasting. We have a summary of the book and interview with him on our podcast (followed by a second interview).
You can see a more thorough review of Tetlock’s work prepared by AI Impacts, with a lot of fascinating data. For instance, the training programme was found to improve accuracy by around 10%, with the effect lasting for several years.
Forecasting isn’t exactly the same as good judgement, but seems very closely related – it at least requires”weighing up complex information and coming to calibrated conclusions”, though it might require other abilities too. That said, I also take good judgement to include picking the right questions, which forecasting doesn’t cover.
All told, I think there’s enough overlap that if you improve at forecasting, you’re likely going to improve your general judgement as well. I don’t cover other ways to improve judgement as much, because I don’t think they have as much evidence behind them.
So here are some ways to improve your judgement:
Being well calibrated is an important input into judgement, and I mention it as part of my short definition of judgement at the start. It means being able to quantify your uncertainty so that if you say you’re 80% confident in a statement, you’ll be right four out of five times.
This is important because there’s a big difference between 20% and 80% confidence, but these could easily both be called ‘likely’ in natural language.
There is evidence to suggest that people can improve their calibration in just an hour of training, and there’s some chance this transfers across domains.
For this reason, Open Philanthropy commissioned a calibration training app, which you can try here.
As with any skill, the best way to improve is to actually practice. To improve your forecasting, you can practice making forecasts – especially if you also start to apply some of the techniques covered in the next section while doing it.
Here are some ways to practice:
One weakness of the research on forecasting is that it doesn’t cover how to focus on the right questions in the first place. This is an area of active research briefly covered in our second podcast with Tetlock.
Keep in mind that having calibrated overall judgement isn’t the only habit of thought that matters. Within a team you may want some people who generate creative new ideas and advocate for them even when they’re probably wrong, or challenge the consensus even when it’s probably right. That may be easier to do if you’re overconfident, so there may be a tension between what habits are best for individual judgement and what’s most helpful when contributing to a group’s collective judgement.
Luke from Open Philanthropy lists a few techniques for improving judgement that have some backing in the research:
Here are Tetlock’s 10 commandments of forecasting (plus one meta-command), as summarised by AI Impacts:
The people with the best judgement also seem to have a certain mindset. Luke again:
According to some of the most compelling studies on forecasting accuracy I’ve seen, correlates of good forecasting ability include “thinking like a fox” (i.e. eschewing grand theories for attention to lots of messy details), strong domain knowledge, general cognitive ability, and high scores on “need for cognition,” “actively open-minded thinking,” and “cognitive reflection” scales.
And here’s Tetlock’s portrait of a good forecaster:
I haven’t seen any research about this, but I expect that – as with many skills and mindsets – the best way to improve is to spend time with other people who exemplify them.
Spending time with people who have great judgement can help you improve almost automatically, by giving you behaviours to model, making it easy and fun to practice, giving you immediate feedback, and so on.
Anecdotally, many people I know say that what they found most helpful in improving their judgement was debating difficult questions with other people who have good judgement.
Although experts are often not the best forecasters, all else equal, more domain knowledge seems to help. Luke from Open Philanthropy:
Several studies suggest that accuracy can be boosted by having (or acquiring) domain expertise. A commonly-held hypothesis, which I find intuitively plausible, is that calibration training is especially helpful for improving calibration, and that domain expertise is helpful for improving resolution.
One of the other lessons of Tetlock’s work is that combining many forecasts usually increases accuracy, and the average of many forecasts is better than most individuals. This suggests that another way to make better judgements is to seek out other people’s estimates, and to sometimes use them rather than your personal impressions.
Exactly how much weight to put on the average estimate compared to your own views is a difficult question.
It’s also important to remember it can often still be worth acting on a contrarian position, so long as the upsides are much bigger than the downsides.
I hope to write more about how to balance judgement and independent thinking in another post (for now, I’d recommend reading In defence of epistemic modesty by Greg Lewis and the comments).
This is a supporting article in our advanced series. Here are some others you might find interesting:
