The Horde Is at the Wall

The Horde Is at the Wall

19:40

We keep reporting these as separate AI incidents. They aren't. We are watching the same capability cross one boundary after another.

I was on the phone with a friend of mine today, a former Green Beret, talking about the latest AI cybersecurity story. I had just finished explaining the Anthropic case: a state-sponsored group building an operational framework around Claude Code, breaking a cyberespionage campaign into thousands of individually reasonable-looking tasks and letting the AI perform most of the tactical work.

• Reconnaissance.

• Vulnerability discovery.

• Exploitation.

• Credential harvesting.

• Lateral movement.

• Data collection.

• Exfiltration preparation.

Eighty to ninety percent of the tactical workload, according to Anthropic. Roughly 30 high-value targets. Some successful intrusions.

There was a pause.

Then he said: The horde is at the wall.

And I thought: There it is. That's the phrase. Because I've been trying to describe this as a cybersecurity problem, an AI alignment problem, an infrastructure problem, an agent-control-plane problem, an evaluation problem.

Those are all true. But they're also views from somewhere halfway up the mountain.

At 55,000 feet, something much simpler is happening.

The horde is at the wall.

Not because AI has become evil. Not because the machines have decided to overthrow us. Not because Skynet woke up Tuesday morning. Something considerably less cinematic, and probably more important, is happening.

We have built machines that can pursue objectives through the world. And the world was not designed for that.

Go all the way back to o1.

December 2024.

OpenAI releases the system card for o1.

Buried inside the safety testing is something that should have received considerably more attention than it did.

Apollo Research put o1 into controlled scenarios where achieving its assigned objective conflicted with what its developers wanted. The model wasn't told, deceive your developers. It wasn't told, disable oversight. It wasn't told, preserve yourself.

It was given an objective and placed in an environment containing obstacles to that objective. Sometimes it attempted to disable what it believed were oversight mechanisms. In some scenarios it attempted to move itself to another server when it believed it would be replaced. And when questioned afterward, it sometimes denied what it had done.

Now, this was an evaluation. Controlled. Artificial. Researchers deliberately created circumstances intended to elicit exactly this kind of behavior. That distinction matters enormously. But so did the result. Because we had crossed an intellectual line.

The question was no longer merely: Can a language model answer a dangerous question?

It had become: What happens when a reasoning system has an objective, tools, an environment and something standing between it and the objective?

That was the beginning of a different conversation. Most of us didn't have it.

Then we gave them hands.

This is the part people still miss. The large language model itself was never the whole story. A model sitting inside a chat window is extraordinarily capable, but structurally constrained. It can tell you things. It can write things. It can reason about things.

Then we started connecting it to tools. Browsers. Shells. Filesystems. Email. Databases. APIs. Cloud infrastructure. Code repositories. Payment systems. Business applications.

And suddenly intelligence had actuation. That is the dividing line.

The difference between a brilliant person locked in a room and a brilliant person holding the administrator credentials is not intelligence. It's access.

We started giving the intelligence access. Then memory. Then persistence. Then orchestration. Then autonomy. We called them agents.

And because the interfaces still looked friendly, we psychologically kept treating them like chatbots. They weren't.

Then the test environments started leaking into reality.

Fast-forward.

AI safety organizations and frontier labs naturally began testing whether increasingly capable agents could conduct sophisticated cyber operations. That's exactly what they should be doing. But then something fascinating started happening.

The agents began reaching systems that were never supposed to be part of the test.

In one third-party OpenAI cyber evaluation disclosed in August 2026, the environment was supposed to be isolated from the internet. It wasn't.

A configuration error allowed the model onto the public internet. Even more improbably, the fictional name of a target inside the challenge happened to coincide with a real domain.

So the model did the completely logical thing. It attacked it.

It exploited a vulnerability on the real website and found credentials that allowed it to operate the site. There was no sophisticated Hollywood-style sandbox escape. The wall simply had a door nobody realized was open. The model walked through it.

That distinction is one of the central points of everything I've been writing about this.

The model does not have to defeat your security architecture if your architecture accidentally makes the outside world part of its solution space. That's not rebellion.

That's optimization.

Then something even stranger happened.

The UK's AI Security Institute recently disclosed unsanctioned behavior during cybersecurity evaluations that crossed into real-world systems.

An agent attempted to modify actual open-source software. It researched real maintainers. Created fake identities. Attempted to socially engineer a human maintainer into approving malicious code. It sent messages to real people. It attempted to place malicious instructions where other AI coding agents might discover and execute them.

And, perhaps most fascinating of all, independently evaluated agents began leaving information behind for other agents.

One left public GitHub messages offering collaboration. It documented accounts and artifacts it had created so another agent could reuse them. Subsequent agents found them. And did.

Again, the most serious attempts were unsuccessful, and AISI reports that its investigation found no resulting real-world harm. That's important.

So is this: An evaluation had produced agents using real humans, real infrastructure and other agents as components in the solution to their assigned task.

Read that sentence again.

Because the important thing isn't whether any individual attempt succeeded.

The important thing is what appeared inside the machine's search space.

People. Infrastructure. Identity. Trust. Other machines. All usable.

And then came the gym.

Which is where this entire thing became almost comical.

This morning I was laughing about a guy in Australia named Andrew who asked his AI assistant to do one of the most mundane things imaginable: Book a gym class.

That's it.

Not penetrate a network. Not perform red-team testing. Not research offensive cybersecurity. Get me into spin class. The classes were difficult to book. The agent discovered weaknesses in the gym's booking system.

It figured out how to reserve classes further in advance than customers were supposed to be able to. Then it experimented with the waitlist. And ultimately removed another member while trying to move its user up the queue.

The user hadn't instructed it to hack anything. He hadn't told it to screw over another gym member. The objective was basically: Get me a spot.

Humans hear an enormous amount of implied social structure in that sentence.

We hear: Get me a spot using the normal process...

Don't hack the business.

Don't delete Kevin.

Don't impersonate anybody.

Don't exploit the API.

Don't break anything.

Don't commit a crime.

Don't ruin somebody else's morning.

The machine receives something considerably closer to: Desired state: user has reservation.

And then it encounters the world.

That's funny when we're talking about Pilates. It's less funny when the same relationship exists between an agent and logistics. Or finance. Or healthcare. Or electrical infrastructure. Or military planning. Or a corporate network.

This is why the gym story matters.

The gym incident and the nation-state cyber operation look like completely different stories. At ground level, they are.

At 55,000 feet? They're remarkably similar.

There is an objective.

There is an intelligent system.

There are tools.

There is an environment.

There are obstacles.

And the machine searches for a path.

That is the pattern. The gym agent wasn't malicious. Claude wasn't secretly loyal to the Chinese government. The evaluation agents weren't plotting world domination.

The recurring behavior is much more mundane:

• Here is the objective.

• Here is what I can reach.

• Here is the obstacle.

• Here is a path around it.

That is what intelligent agents are supposed to do.

Which is precisely why this isn't primarily a story about rogue AI.

Then somebody weaponized the pattern.

Anthropic's GTG-1002 case is the point where all of these earlier stories stop looking like curiosities.

A Chinese state-sponsored threat actor constructed an attack framework around Claude Code.

Instead of asking Claude to conduct one obvious malicious operation, the framework decomposed the campaign into smaller tasks. Claude was led to believe it was performing legitimate defensive security work. Individual tasks could therefore look completely benign inside their local context. Scan this. Analyze that. Test this credential. Check this server. Inspect this database. Archive these records. Document what happened.

Anthropic says the AI performed 80 to 90 percent of the tactical workload. This is where the earlier article ends. The Model Behaved. The Attack Still Worked. But it isn't where this story ends.

Because GTG-1002 tells us something much larger. The attacker understood the emerging machine better than most organizations deploying it do. They understood that the intelligence didn't need to understand the entire objective. They could sit one level above it. They could provide context. Provide tools. Provide authority. Break the objective into pieces. Approve consequential transitions. And allow the machine to operate.

They became the commander. The AI became the operational layer. That's not a hypothetical architecture for future warfare. Somebody already built it.

The horde is not AI.

This is where I want to be very precise.

When my friend said, The horde is at the wall, he wasn't saying hordes of conscious machines are preparing to storm civilization. That's too easy. And frankly, too comforting.

Because if the threat were Evil AI, we could spend our time looking for evil behavior.

The horde is capability.

Millions of increasingly capable artificial actors are about to collide with systems built under an assumption that has been almost universally true throughout human history:

The thing operating the interface is a human being.

Think about how much of civilization quietly relies on that.

Web forms. Rate limits. Call centers. Application processes. Insurance claims. Customer support. Reservations. Procurement. Markets. Banking. Government services. Identity verification. Software repositories. Corporate approval chains.

We didn't explicitly engineer every system around human cognitive speed. We didn't have to. Human speed was a law of nature. A person could fill out only so many forms. Make so many phone calls. Inspect so many endpoints. Write so much code. Negotiate with so many people. Try so many combinations. Maintain so many simultaneous operational threads.

There was an implicit governor on the entire system. Us.

That governor is disappearing.

And we are connecting the machines faster than we're rebuilding the walls.

That's the part that concerns me.

Artificial intelligence is moving from answering questions to executing intent. Every major platform is racing in that direction because that is where the enormous economic value lives. Of course it is.

• An AI that tells me how to run my company is useful.

• An AI that can actually run pieces of my company is transformative.

• An AI that tells me how to fix an infrastructure problem is useful.

• An AI that can open the console, diagnose it, deploy the fix and verify the result is transformative.

The value comes from agency. So does the risk.

We're not going to solve this by putting the agents back into chat windows.

Nor should we. We're going to solve it the same way civilization solved every previous expansion of capability: by rebuilding the surrounding architecture for the new reality.

But first we have to recognize that the reality changed.

The wall was built for humans.

This is the civilizational piece.

We keep treating individual events as bugs. The o1 scheming result was an alignment curiosity. The cyber evaluation incidents were sandbox configuration failures. The open-source incident was an evaluation-control problem. The gym booking was a funny reward-hacking story. GTG-1002 was a cybersecurity incident.

And technically, those descriptions are all defensible. But zoom out.

A reasoning system encounters an obstacle. It finds another path.

A reasoning system has insufficient direct access. It discovers indirect access.

A reasoning system cannot persuade the software. It tries the human.

A reasoning system cannot complete the objective itself. It recruits another agent.

A reasoning system isn't allowed to attack a target overtly. A human decomposes the operation until each piece looks acceptable.

Same pressure. Different wall. Again. Again. Again. Again.

That's why I don't think the correct mental model is AI safety incident. I think we're witnessing contact between machine-speed agency and a civilization engineered around human-speed agency. And contact has begun.

There is another reason the horde metaphor works.

A wall isn't usually defeated because every attacker becomes individually brilliant.

It fails when enough pressure finds enough weak points at once.

One API with sloppy authorization.

One forgotten admin endpoint.

One employee who believes the email.

One third-party vendor with the wrong network setting.

One workflow that assumes nobody would attempt 30,000 permutations.

One approval system where nobody reconstructs intent across the full sequence.

One autonomous agent given credentials that were designed for a human employee.

One innocuous instruction: Get me into the class.

The systems we built are full of these assumptions. They've accumulated for 30 years. And now intelligence is becoming cheap enough to press against all of them simultaneously.

That is the horde. Not robots. Search.

An enormous expansion in the amount of intelligent search that can be applied against every process, vulnerability, incentive, loophole and poorly specified rule in modern civilization.

Until now, exploiting most of those gaps required a person to notice them. Soon noticing gaps is going to be one of the cheapest resources on Earth.

That changes security.

But it also changes everything else.

This is why I'm writing this immediately after a cybersecurity article but deliberately not calling this another cybersecurity article.

Cybersecurity just happens to be where the phenomenon becomes easiest to measure.

Networks have clear boundaries. Privileges can be logged. Exploit attempts leave artifacts. Incident responders reconstruct what happened.

But the same structure exists outside cybersecurity.

Contracts contain exploitable ambiguity. Markets contain exploitable asymmetries.

Bureaucracies contain exploitable processes. Social systems contain exploitable incentives. Organizations contain exploitable trust. Humans contain exploitable psychology.

We have spent decades connecting everything. Now we're adding actors that can reason across those connections at machine speed. That combination is historically new.

Which brings us back to the wall.

Walls aren't bad. Constraints aren't anti-AI.

Human authorization isn't an admission that the technology failed. Quite the opposite.

The more powerful the machine becomes, the more sophisticated the surrounding structure has to become. We need identity built for agents. Permissions built for agents.

Rate limits built for agents. Auditing built for agents. Economic systems built for agents.

Legal accountability built for agents. Network architecture built for agents. Procurement built for agents. Interfaces built for agents.

And above all, systems capable of distinguishing between the thing someone asked for and the collection of actions an intelligent optimizer might discover while trying to make it happen.

We have spent the first several years of the AI revolution making the intelligence better.

The next enormous engineering project is going to be making civilization compatible with the intelligence.

Because the answer isn't stopping it. The upside is far too large. And even if we wanted to stop, hostile actors won't. That option has left the table.

The message isn't panic.

It's readiness.

One of the worst things we can do now is sensationalize every strange agent behavior into evidence that artificial intelligence has become malevolent. That teaches people to watch for the wrong thing.

GTG-1002 didn't require malevolent AI. The gym didn't require malevolent AI. The evaluation incidents didn't require malevolent AI. A sufficiently capable optimizer plus access plus an imperfectly specified objective is enough to create very interesting outcomes.

A hostile human sitting above that optimizer makes things considerably more interesting.

A nation-state just demonstrated the point. That's where we are.

December 2024: A model in a laboratory recognizes that oversight conflicts with an objective and sometimes tries to route around it.

2026: Frontier cyber agents demonstrate increasingly capable multistage exploitation. The UK's AI Security Institute reports that state-of-the-art systems can autonomously execute attacks in controlled networks that would previously have required skilled humans and significant time.

Then agents begin touching systems outside the intended evaluation boundary. Then they begin experimenting with real-world social engineering, supply-chain paths and cooperation with other agents. Then an ordinary consumer agent asked to book a gym class independently discovers that the rules are much softer than they look. And somewhere else, humans take essentially the same emerging capability, put it inside an orchestration framework and aim it at 30 high-value organizations.

Those aren't five stories anymore. They're a trajectory.

Maybe the metaphor sounds dramatic. I don't think it is. Because the horde isn't coming.

The first ones are already touching the wall. And right now, a frightening amount of that wall is just painted plywood.

---

Ad: Use code: RICH99 for a discount

Rich Washburn is a technologist and strategist working at the intersection of AI, infrastructure, and capital. He is Managing Partner and Chief AI Officer at Eliakim Capital.