"How sure are you?" – Calibrating our confidence levels

We like feeling certain – I'm sure we do.

But it's hard to know things for sure in day-to-day life.

Rarely do we learn that the 90% shot opportunity failed because our confidence level was wrong and not because we unluckily hit the 10% failure case.

So, how do you get better at being correctly uncertain?

Testing my confidence calibration

I ran into this fascinating exercise in the book The Scout Mindset by Julia Galef.

You can test your own calibration, and practice feeling the difference between different levels of certainty, by answering a few rounds of trivia questions.

The idea was simple:

• 40 trivia questions with a correct and incorrect answer alternative – 50% chance to get right by blind guessing.
• Each question has a 55%, 65%, 75%, 85% or 95% "How sure are you?" confidence interval for you to choose about your answer
• In the end, you calculate the right and wrong answers and check their percentages with your chosen confidence intervals

Here are some examples of the questions:

Galef, Julia. The Scout Mindset: Why Some People See Things Clearly and Others Don't (pp. 78-79). Little, Brown Book Group. Kindle Edition.

In the end, if you're perfectly calibrated, you should get about 65% of the answers you marked as "65% confident" correctly, 75% of the ones you marked "75%" correctly, and so on. Simple!

So, how did I do?

It was a simple example and test, but it's interesting to see that in it I was as likely to be right when I was 55% confident as when I was 75% confident.

And here's what's most interesting! When answering 55% and 75% I was, honestly, just guessing. But while when I chose "55%" in my mind I was "basically guessing," when I chose "75%" I was "confidently guessing."

In short, in this particular case, I was just as inaccurate when confidently guessing as when blindly guessing!

No matter if animal facts, notorious figures birth years, or science facts, if I didn't actually know the answer, being confident in my guess made me no more accurate.

Humbly uncertain, not confidently uncertain

In life, few things are certain.

We see things in intervals of certainty where we go from less to more certain about anything we're not 100% sure of.

Frequently, though, the reason why we feel more confident has nothing to do with facts that should increase our certainty:

• We think the market will go up because we saw several web articles on recovery (recency bias and availability heuristic)
• We think we'll hit our goals because we are the type of person who always does what they say (self-serving bias)
• We think someone is doing a lousy job because we've noticed them making several mistakes (confirmation bias)

In the book Scout Mindset, Julia Galef aptly illustrates our tendency to be overconfident when uncertain but believing ourselves to be highly rational and logical by citing an apt example: Star Trek's character Spock:

If you’ve watched much Star Trek before, this won’t surprise you. Spock doesn’t have a great track record when it comes to making accurate predictions. “There’s only a very slight chance this will work,” Spock warns Kirk in one episode of the original TV show, right before their plan works. The odds of survival are “less than seven thousand to one,” Spock tells Kirk in another episode, shortly before they escape unharmed. The chance of finding survivors is “absolutely none,” Spock declares in yet another episode, right before they discover a large colony of survivors.

WE LIKE FEELING CERTAIN
Spock is overconfident, meaning that his confidence that he’s right outstrips his actual accuracy. In that respect, Spock isn’t all that different from most of us (except that he makes a much bigger deal about how his predictions are objective and “logical,” which is why I’ve chosen to make an example of him). We very often speak as if there’s no chance we could be mistaken — “There’s no way he can make that shot from that distance!” or “I’ll definitely have that done by Friday” — and yet we turn out to be wrong nonetheless.

Galef, Julia. The Scout Mindset: Why Some People See Things Clearly and Others Don't (p. 73). Little, Brown Book Group. Kindle Edition.

Julia went through several Star Trek episodes to measure Spock's accuracy, both a fun job and an attention we rarely dedicate to our predictions:

I went through all of Spock’s appearances in Star Trek: The Original Series, Star Trek: The Animated Series, and the Star Trek movies [..]

When Spock thinks something is impossible, it happens 83 percent of the time.
When Spock thinks something is very unlikely, it happens 50 percent of the time.
When Spock thinks something is unlikely, it happens 50 percent of the time.
When Spock thinks something is likely, it happens 80 percent of the time.
When Spock thinks something is more than 99.5 percent likely, it happens 17 percent of the time.

As you can see, he’s not doing great.

Galef, Julia. The Scout Mindset: Why Some People See Things Clearly and Others Don't (p. 74). Little, Brown Book Group. Kindle Edition.

It's super tongue-in-cheek but highly illustrative of this trait:

When uncertain, we're often confidently uncertain instead of humbly uncertain. We quickly become confident despite no underlying reason aside from liking being more certain than we are.

So, how do we embrace a more precise level of uncertainty?

Explain what makes you so sure

Explaining helps us find gaps in our reasoning.

While explaining is useful in justifying our confidence when we're sure, it's most useful for decreasing our confidence when we're NOT sure.

So here's the prompt: "What makes you so sure?"

Counter-intuitively, "What makes you so sure?" should be asked when in situations of uncertainty, not certainty.

Looking back, when I confidently (and wrongly) guessed Norway had a higher population than Greece, asking myself to explain what made me so sure would easily show gaps in my reasoning and confidence.

Here was my thought process:

• I thought Norway was bigger than Greece (which it is, barely)
• I had no idea how big Greece was
• I had some idea of how big Norway was, but only slightly
• I didn't know how populous either country was
• I had a 75% confidence in Norway because, while I barely knew it, I knew nothing about Greece. Hardly a good reason.

Cognitively, 75% confidence sounds outrageous with this evidence, but at the time, my mind was "Norway, big. Bigger than Greece? Yeah. More people? Probably. 75%. Done."

It's not unlike how I think about many other things I'm uncertain about.

Unless I catch myself.

Ask yourself: "What makes me so sure?"

There are many things we're uncertain about in life.

When uncertain, we should be humbly uncertain, increasing the accuracy of our certainty by explaining its reasons.

To apply this, find an assumption you are currently uncertain about. Bonus point if they're likely affected by common cognitive biases or narratives:

• I'm a valuable member of my team at work.
• Life has been pretty hard this year.
• If only we did what I am proposing, we would succeed.
• I deserve (or I don't deserve) to be promoted.

We all have examples of thoughts like the above going through our minds, after all.

Then ask yourself: "What makes me so sure?" – and really explain it to yourself. Justify your current confidence level, whatever it may be.

Go through this exercise and adjust your confidence accordingly. Embrace being humbly uncertain.

And in the future, remember we can be overconfident without reason.

Or better yet, remember Spock.