The Ceremonial Approval Layer
- Rich Washburn
- 5 hours ago
- 3 min read
In an interview with The Economist's Zanny Minton Beddoes, Musk said he expects AI to exceed the combined intelligence of humanity in roughly five years, and that looking ten years out, humans probably won't meaningfully be "in control" of something vastly more intelligent than themselves. He used the human/chimpanzee analogy: if the intelligence differential becomes enormous, expecting the less-intelligent species to remain in charge is difficult to justify.
And yes, the 10–20% extinction-risk number is real. That's his previously stated estimate, which came up in connection with the interview. But there's a distinction I think gets absolutely mangled in that graphic: Musk isn't really predicting "Skynet takes control in 2036." He's describing something closer to control becoming an incoherent concept once the intelligence asymmetry gets large enough.
That's much more interesting.
Think about what "human control" currently means. We specify the objective. We decide when the system runs. We authorize actions. We control compute, electricity, credentials, networks and physical actuators. Ultimately there's a human somewhere above the machine in the authority hierarchy.
Now imagine a system that isn't merely smarter than one human, but can perform almost every economically useful cognitive task better than essentially all humans collectively.
At some point:
Are humans controlling the AI, or are humans merely occupying the ceremonial approval layer of a civilization being intellectually operated by AI? And that is where I think Musk's observation deserves more attention than his date. Because we don't need AGI, consciousness, rebellion or a robot army for that transition. We can voluntarily hand over control one rational decision at a time.
AI diagnoses better → let it diagnose.
AI allocates capital better → let it allocate.
AI designs infrastructure better → let it design.
AI writes software better → let it write.
AI manages logistics better → let it manage.
AI makes military decisions faster → let it advise.
AI runs companies better → let it operate them.
Every individual delegation makes perfect sense. And eventually somebody looks around and realizes that humans technically retain authority over everything while actually making almost none of the consequential decisions.
That's the scenario I find considerably more plausible than "AI escapes."
It also connects directly to the agency problem I've been circling: the dangerous threshold may not be when AI takes agency from humanity. It may be when humanity discovers that outsourcing agency was incredibly convenient.
The 2031/2036 dates? I wouldn't put much weight on them. Musk has a legendary tendency to attach dates to technological trajectories that prove directionally insightful and temporally... adventurous. But the underlying question?
That's the question.
And bizarrely, extinction may not even be the most useful thing to worry about. There's an entire spectrum between humanity controls AI and humanity is extinct called human disempowerment. Researchers are increasingly examining exactly that possibility: progressive deskilling and delegation until relying on human judgment stops making economic or institutional sense.
That's much closer to the future I would watch for.
Not: "When does AGI arrive?"
But: "At what point does keeping the human in the loop become the objectively worse-performing option?"
Because when that happens, we're going to learn whether "human in the loop" was a safety architecture...
...or merely a temporary luxury afforded to us while we were still the smartest thing in the loop.
Rich Washburn is a technologist and strategist working at the intersection of AI, infrastructure, and capital. He is Managing Partner and Chief AI Officer at Eliakim Capital.
Comments