Hand of God

Superintelligence won't just be smarter than us. It'll make moves we don’t understand because we can’t understand.

And we'll follow them.

Move 37

In March 2016, DeepMind’s AlphaGo played Lee Sedol, one of the best human Go players, in a five-game match.

In game 2, on move 37, AlphaGo placed a stone on the fifth line that no professional would play. The commentators thought it was a mistake. Lee Sedol got up and left the room for about fifteen minutes. AlphaGo’s own estimate was that a human would play that move about 1 in 10,000 times.

Of course it wasn’t a mistake, we just couldn’t understand it. And it won the game. 10 years ago now.

Fan Hui, the European champion who had lost to AlphaGo a few months earlier, said: “It’s not a human move. I’ve never seen a human play this move. So beautiful.” Lee Sedol said after the game that he’d thought AlphaGo was just probability calculation, “But when I saw this move, I changed my mind. Surely, AlphaGo is creative.”

Go players have a name for the perfect move, the one only a god could find: kami no itte, the hand of God. Curiously, in that match the move known as hand of God was Lee Sedol’s move 78, which guaranteed his one victory against AlphaGo in the 1–4 series before his retirement.

Move 37 is what superintelligence will look like. A move that looks wrong, that the best of us can’t explain.

But a move that works, and that all players will just assume should be played.

Taking the cat to the vet

When someone takes their cat to the vet, there’s no amount of explaining that will make it understand why it’s going.

And it’s not about language. Even if your cat understood every word you said, it wouldn’t help. The problem isn’t that the cat can’t understand your words. It’s that it can’t understand veterinary medicine.

From the cat’s point of view, here’s what happens: you put it in a box, drive it somewhere that smells terrible, a stranger (or Tia Maira in our case) holds it down, sticks it with a needle, and it wakes up later shaved and cut open.

That’s all the cat will ever know.

Same with babies and the doctor. You can’t explain vaccines to a six-month-old. Not because you’re bad at explaining, but because understanding the explanation requires a brain the baby doesn’t have. All they get is the needle and the pain.

Strategically, we are the babies. It’s impossible for us to find move 37, and when we see it we can’t understand it.

More and more, we’ll be handed things we have to accept on faith. Not because they require faith, but because we have no other choice but to trust AI like a cat trusts its owner when going to the vet.

Proofs that aren’t written for humans

It’s hard for us, used to being the most intelligent entities on Earth, to fathom what it’s like to be orders of magnitude more intelligent. But perhaps the recent AI progress on math can help us.

A month ago I wrote about OpenAI dropping 722 math manuscripts on open problems in a single week [Link]. A commenter mentioned “but many of the manuscripts haven’t yet been verified by humans,” unaware he was making my point for me. So let’s look at some more specific examples.

In September, OpenAI claimed an unreleased model had proved finite-time blowup for the Navier-Stokes equations (with external forcing), one of the Clay Millennium Problems. Look at what the mathematicians said when they tried to read it. Oxford’s James Maynard: “So far it’s been very difficult to really extract any human understanding from this new AI proof.” Brown’s Javier Gómez-Serrano was more direct: “The paper is not written for humans.”

The same month, Anthropic announced that Claude had formalized Fermat’s Last Theorem in Lean, a language computers can check. Dozens of agents, working in parallel for 11 days, also proving more than 30,000 supporting theorems along the way. 13M lines of Lean that certainly don’t fit in any book’s margin.

Scott Aaronson, one of many I could quote, talks about the conundrum the role sees itself in now given AI.

What are the new rules for when you get to write a paper with your name on it, and, y’know, get credit for it? That you fully understand the proof, can give talks about the proof, can answer questions about it, take responsibility for its correctness? Do you need to have played any role in finding the proof?

Aaronson, Scott. The Age of Wonders and Terrors [Link]

That’s the cat at the vet. We know it worked. We have no idea why.

Not because we haven’t yet understood it.

But because we are incapable of it.

Baby medicine

We’ll be tempted to try to keep the frontier at the edge of human understanding. Only accept proofs we can follow. Only deploy what we can explain.

That’s like studying medicine but keeping it explainable to a baby. We won’t do it. The incentives are just too strong not to.

If one lab keeps its AI to human-understandable math, and another doesn’t, the second one proves the theorems, cures the diseases, and wins the contracts. The prisoner’s dilemma dominant strategy I wrote about applies here too, and all entities will play the dominant strategy.

Given the choice between understanding and winning, we’ll choose winning; not personally, but economically. Literally the winners will stay while the Luddites get cast aside.

And no amount of pooling human knowledge can survive this competition.

Me, me, me

Today, the best way to solve really hard problems with humans is to get a lot of humans together.

Nobody understands everything that goes into Google, or Apple, or OpenAI, or Anthropic. Each of these companies is a big pile of human brains coordinating on something no single brain could do alone. That’s the best collective intelligence we’ve come up with so far, AI excluded.

But it isn’t very efficient. Our brains are limited, and so are the interfaces between them. We talk, we message, we write documents, we have meetings (of varying levels of effectiveness). If you think of a company as a distributed system, it’s a terrible one: slow network, lossy protocol, and nodes that become unavailable at 6pm and erase part of their data by Monday.

Humans are bad at sharding and bad at networking.

AI, on the other hand, is amazing at both. Dozens of agents proving 30,000 theorems in parallel and stitching them into one proof in 11 days isn’t just a testament to raw intelligence. It’s a testament to the breakthrough of transformers and parallelization.

Besides, this isn’t Pluribus. Humans are selfish, and we keep tying understanding to what a single person can hold in their head. That’s what “I understand it” means: I, one person, can follow it. Because that’s what we personally care about. Me, me, me.

Intelligence that has to fit inside one human head or be transmitted through our lossy protocols is very limited when compared to all potential intelligence we can produce through networked GPUs, TPUs, Neural Engines, and whatever comes next.

It’s not even close.

The more powerful will win

So here’s where I think this goes.

We’ll continue to make bad decisions, like we do now. Of course. Except now AI will disagree. It’ll tell us why. And we won’t understand the explanation.

Then what?

Then the more powerful side wins. Today, that’s us. In time, that’ll be AI.

You misunderstand me, it was not a request

Alan Moore saw this coming in the ’80s.

In Watchmen, Dr. Manhattan is the only character with actual superpowers. A lab accident turns Jon Osterman into something that can see time, rearrange matter, and be in several places at once. The book doesn’t treat him as a superhero. It treats him as a god, and slowly, as something that stops being able to relate to humans at all. For a comic from 1986, it’s an amazingly prescient picture of superintelligence.

There’s a scene in 1977, during riots against the costumed heroes. A hostile crowd gathers outside the White House. Someone points at Manhattan and yells “It’s against God!” Laurie is pulling the ringleaders out one by one, but it’s too slow. So Manhattan says:

“Pay attention. You will all return to your homes.”

Someone in the crowd shouts back: “Oh, yeah? And what if we don’t?”

“You misunderstand me. It was not a request.”

The next day, I am reading in the paper of two people who suffered heart attacks upon suddenly finding themselves indoors. More would have suffered during a riot, I’m certain.

Moore, Alan and Gibbons, Dave. Watchmen, Chapter IV (p. 22).

Dr. Manhattan isn’t a villain doing evil. It’s a more powerful mind doing the math, getting it right, and not needing anyone’s permission to follow through.

The people in the crowd don’t get a vote. They don’t even get an explanation.

They just get sent home.

And that’s already the theme of how we operate with AI.

I believe you

ChatGPT, Claude, Codex, Google Maps, search, social media, from Netflix suggestions to weather forecasting: We already blindly trust AI on lots of things.

Soon enough, it’ll be dictating all our moves, and we’ll be disagreeing at our own peril, assuming we still can. Self-driving cars, identity, credit, as time goes on, more and more of our lives become digital and influenceable, or controllable, by AI.

The hand of God controlling not only some moves, but every move on the board.

Humans relegated from moving in any direction to moving only forward and taking on the diagonal.