AWS 311-DOWN-DOWN

Rich Washburn
Oct 20
3 min read

When us-east-1 Sneezes, the Internet Gets a Cold

At exactly 3:11 AM this morning—because apparently the cloud has a flair for irony—Amazon Web Services’ us-east-1 region tripped over its own DNS resolver and faceplanted, taking half the internet down with it.

For those of us of a certain age, sipping our morning coffee while watching dashboards fail to load, there was only one thing going through our heads:

🎶 Gonna take the internet down… down… 🎶

Yes. 311. The band. The timestamp. The vibe.

It was the kind of moment where tech meets nostalgia and neither one is stable.

🧱 A House of Cards Named DynamoDB

The root cause? A DNS resolution failure affecting the DynamoDB API endpoint.

If you’re not familiar with DynamoDB, just imagine that it’s the central nervous system for dozens—maybe hundreds—of other AWS services. IAM, Lambda, Step Functions, Cognito, EventBridge—you name it. All of them depend on DynamoDB in some capacity. And all of them are clustered, lovingly, around us-east-1, the Jenga base of the modern internet.

When that fails?Everything above it wobbles like a cartoon anvil in midair.

🌍 Wait, How Is This Global?

This wasn’t just a “some folks had a rough morning” type of event. This was global.

Affected services included:

Lufthansa (hope you weren’t trying to board a plane),
Airbnb (or book a place to sleep),
Snapchat (can’t snap about it),
Perplexity, Signal, Canva, Granola, Substack, Coinbase, Airtable, npm, DockerHub, Atlassian, Slack...

Some companies couldn’t even update their own status pages… because those pages depended on the services that were broken.

It's like trying to call 911 only to discover your phone app is 911.

The Single Point of Failure That Wasn’t Supposed to Be

Cloud computing was supposed to solve this. It promised:

✅ Redundancy

✅ High availability

✅ Geo-failover

✅ Resilience

And yet, here we are—again—watching us-east-1 take a knee and bring the internet down with it like it’s the final Jenga block holding up the digital skyline.

In theory, cloud decentralization should make infrastructure failures isolated. In practice, we’ve built hyper-dependence on a single region... with global consequences.

It’s not decentralization. It’s centralization with multi-AZ optimism.

The Frustration Hits Home

Personally? I just needed to:

Finish a presentation,
Book a flight,
Lock in accommodations.

Instead, I was bouncing between downed services like a digital pinball machine. Because every one of those tasks relied on some part of the AWS ecosystem—directly or indirectly—living in the exact spot that was on fire.

This Isn’t the First Time

Let’s be clear: this isn’t new. us-east-1 has become notorious for being both essential and fragile. It’s the default region for most developers, startups, and even major platforms. Which means…

When it goes down, so does your app, your analytics, your auth tokens, your CI/CD pipeline, and maybe your patience.

And while AWS is better than most at post-mortems and recovery, the bigger issue isn’t just that it fails. It’s that so much still depends on it.

So What Do We Do?

Let’s not pretend we’re going to fully de-risk the cloud anytime soon. But we can get smarter:

Architect across multiple regions. Stop treating us-east-1 like the center of the universe.
Decouple critical services wherever possible.
Test your failover—not just your backups.
Don’t put all your IAM/Dynamo eggs in one east-coast basket.

Also, maybe reconsider building your core infrastructure on a stack that’s one DNS hiccup away from collapse.

A Perfect Visual Metaphor

This cartoon has been circulating all morning—and it might be the most accurate architecture diagram of 2025:

“AWS us-east-1” — the single stick propping up half the internet.

Final Thought: You Can’t Spell Resiliency Without LOL

As technologists, we love to preach resilience, failover, uptime, five-nines.

But every now and then, the internet throws us a reminder that no amount of infrastructure diagramming can save you from a bad DNS update in Virginia.

And when that happens?

Well...