top of page

The Pandemic IT Rehab Road Trip: Better Late Than Never

ree

You ever hear a friend describe an IT mess so vivid you can smell the burnt plastic? A buddy called me recently, pacing through the kind of “rehab” where the network diagram is a rumor, the backups are a folk tale, and the last guy left behind a Post-it that says “DO NOT REBOOT (seriously).” Halfway through his play-by-play, I started laughing—not at him, but at déjà vu. It was the exact flavor of chaos my crew and I waded into at the dawn of COVID. I realized I’d never written it down: the road miles, the midnight installs, the fixes that should not have worked but did. So here it is—late to the party, sure—but still hot.


This is the field journal from one of the wildest IT rehabs I’ve ever run. Buckle in. You’re riding shotgun.


The Crew

Sean, the stalwart who’d seen enough to be suspicious of hope. Me—six to eight weeks on the job, still wearing the “new badge smell.” Andy, my longtime friend parachuting in as CTO, New York grit in Florida humidity. And Kirkley—Pennsylvanian by mail, mystery by nature; we adopted him mid-tour like a stray who knew how to rack a switch.


We inherited a kingdom of duct tape: decades of “just make it work,” tribal knowledge buried with the people who left, and a change log written in disappearing ink. We weren’t upgrading a stack; we were exorcising one.


Then the world shut down.


Before COVID turned airports into museums, we’d already pulled the pin on big changes: collapsing a hydra of phone systems into one cloud platform (Jive / GoToConnect), unwinding MPLS and OC3 anchors, killing contracts like we meant it. Half our environment was mid-transition when the travel bans hit. Plan A had stages, swim lanes, and a neat Gantt chart. Plan B had four people, a rented Jeep, and a promise: “Charlie Mike”. (Complete Mission)


If we couldn’t fly, we’d drive. If we couldn’t be in the building at noon, we’d be there at midnight. And if we couldn’t buy a server, we’d build a data center out of stubbornness and aluminum.


Stop One:

Waynesboro, Georgia — Recon at the Green Monster

Imagine a press the size of a semi slamming sheets into commercial cookware that ends up on Navy contracts. They called it the Green Monster. The nearby oven ran around 1900°F—stand next to it for five minutes and you develop a farmer’s tan and a new respect for metallurgy.


Buried in the corner: a COBOL mainframe last serviced when shoulder pads were fashionable. The keypad on the door looked like a puzzle for toddlers: 1-2-3-4-#. You could read the code from the wear pattern. Security by exfoliation.


We didn’t rip and replace. Not yet. This was reconnaissance: inventory the zoo, stage phones, drop hardware, and map the landmines. We’d be back to virtualize that COBOL dinosaur later. For now, we took notes, took pictures, and took the hint: this project wasn’t a sprint; it was a siege.





Stop Two:

Avondale, Pennsylvania — Mushroom Country, Metallurgy Class

Avondale supplies an indecent percentage of the mushrooms your grandma cooked into casseroles. The air smells like a soup can with opinions.


Inside, though? Surgical steel—literally. Cutlery brands you’ve bought without noticing the logo. Electrolysis. Annealing. Tempering. Sharpening. A manufacturing ballet with sparks.


Their IT was… almost modern. Multiple brands, multiple sites, a topology that had ambitions. We staged phones, built inventory, and—mid-chaos—onboarded Kirkley. He sat down, looked around, and I said, “Welcome to the circus. Grab a tent pole.”



Stop Three:

Harrisburg, Pennsylvania — The Farmhouse Help Desk

You know that origin story where a founder prototypes blenders in a garage? Now imagine the company converting the actual childhood home into a customer support center. Desks in bedrooms. Headsets in the dining room. KPIs on the fridge.


The basement held a chef’s kiss of duct-tape IT: the phone system tucked under a leaky pipe, mummified in a plastic bag. It overheated and rebooted every couple of hours. “It just does that,” someone said, the IT equivalent of “He’s not biting; he’s playing.”


We staged phones, grabbed barbecue, and a stranger handed me a spice blend I used on everything for the rest of the trip. Ten out of ten, would season again.



RACT — the 80-MPH Remote Autoconfig Tool

Remember that first pandemic week when “work from home” meant “unplug your desktop, guess which cable is which, and good luck”? Our Weston office was a desktop zoo, the entire IT crew (us) was on the road, and supply chains were playing keep-away with laptops. People were firing up their kids’ Windows machines, panic-buying whatever Best Buy still had on a shelf, and calling us to narrate their cable archaeology over speakerphone.


So, somewhere between states—Andy happily driving, me with a laptop balanced on my knees—we built RACT: the Remote AutoConfig Tool. It was the difference between “we’ll get to you next week” and “click this link and get back to work.”


If they could reach a Windows desktop and open a browser, they were five minutes from “lost” to “logged in.”


RACT turned chaos into a self-serve motion. People hit the link, ran the tool, got their stack, and went back to work. Meanwhile, my phone pinged each successful username like a little victory bell.  RACT wasn’t fancy. It was fast, obvious, and relentlessly useful—the exact tool you build when the world is on fire and your only superpower is shipping something that works right now.


Erie

Where Servers Go to Freeze


The Drive In: Ice, Buggies, Red Heads

January in Pennsylvania is a personality test. The Jeep turned into a popsicle between gas stations. Amish buggies handled four-way stops with more discipline than most commuters. We passed a guy sleeping in his buggy while the horse obeyed traffic law like it had a CDL.


We learned about rumspringa and met a pair of redheads who reminded us we were the cultural exchange program. If you’ve never been propositioned between data center stops, were you really doing pandemic IT?




COVID Politics Meets Border Hopping

Pennsylvania declared us “non-essential.” Going in risked fines or worse. So we crossed into Ohio, adopted a bourbon bar on Lake Erie with a leg lamp from A Christmas Story, and waited it out with brisket chili fries that required heavy equipment to eat. The owners charged three bucks a pour and kept saying, “Try this one.” We did!




The Meat Locker Data Center

Back in Erie, their ticket said “heat issues.” What they had was a freezer. It was colder inside the server room than outside on the lake. The problem wasn’t the thermostat—it was history. Every server wore a blanket of dust thick enough to knit. Fans couldn’t spin. Heat sinks were fossilized. We performed archaeology on Compaq ProLiants, finding cockroach confetti and decisions that aged badly.


It wasn’t a server room; it was an IT graveyard—CRTs, dead UPSes, cables braided like a macramé class. Two days of cleanup just to reach the racks. By the time we cracked them open, the verdict was simple: they’re not overheating; they’re suffocating.




ree

“So… We’re Going to Run the Office on a MacBook.”

COVID supply chains meant real hardware was a three-week mirage. We needed to resurrect a full Windows environment now—domain controllers, file shares, SolidWorks CAD—the works.

ree

So we committed a beautiful, practical heresy: virtualized the entire stack onto a high-end Intel MacBook Pro. We slid it onto a rack shelf, plumbed it into the network, and christened the whole contraption with Apple stickers: the Mac Rack.


One of local engineer guys stared at me for a full minute, waiting for the punchline. There wasn’t one. The laptop carried the office until proper gear arrived. Was it absurd? Absolutely. Was it reckless? Measuredly. Did it work? Like a champ.


Lesson: Perfect is a luxury. Available is a strategy. Stability is the goal.


The Ghost Wing

Half the building was condemned—time capsules behind caution tape. Wood paneling. Floral wallpaper borders. A brass lamp still plugged in. Chalkboards in wooden frames. An Avaya One phone like an artifact. Someone had literally mounted a floppy drive to a wall. 1979 didn’t fade here; it paused.


Next door the modern plant thumped along while we threaded cat-6 through yesterday’s ghosts. If you ever want to understand legacy systems, walk through a condemned wing of a factory at midnight.



Hotel Labs & Midnight Installs

Because “non-essential,” we often couldn’t work in daylight. So we turned the hotel into a lab. Three rooms, servers on beds, switches on dressers, Ethernet cables snaking under doors. We’d rack, configure, test, then go full raccoon—sneak into the building after midnight and install.

One night we left the front door unlocked. The cops showed up. We explained no burglar steals servers in January. They laughed, we laughed, and we all agreed to pretend this never happened.


Lake Erie, Briefly

We insisted on seeing the lake. It insisted on seeing us shiver. Eighteen minutes in a heated jacket for me; everyone else bailed earlier. Beautiful, brutal, done.


Erie in one sentence: dust-choked servers freezing to death, a data center running on a laptop, condemned offices you could time-travel in, and bourbon across the border when the state said “No.”


Paris, Kentucky — Viral Load

Kentucky felt like a parallel universe. Restaurants open. Bartenders unhurried. I got sick—COVID? Maybe—and a place called Trackside fixed me with a steaming glass of bourbon dressed with orange, cinnamon magic shit. I went to bed a radiator and woke up human.


Then came the curveball: ransomware. A $20 million Bitcoin demand. Lawyers in a huddle. Execs weighing options. IT on hold. I hate being on hold.


My background includes digital forensics, so I peeked at “encrypted” files. Real crypto is compute-heavy—slow, loud, expensive at scale. This looked… light. The headers were wrong—four junk characters glued to the front. The rest of the file looked pristine. I deleted the bogus bytes and the file opened like nothing happened.


We wrote a script to strip the junk headers at scale, restored the data, and got back to work. No payment. No PR crisis. Just a cleanup.


Pro tip: When ransomware strikes, verify. “Encrypted” is often theater. If you open the file and it’s not high-ASCII soup all the way down, check the header. Sometimes the dragon is cardboard.


What Actually Changed

  • Unified Comms Saved the Day: Moving to cloud telephony before lockdown meant business continuity wasn’t a fantasy. Routing rules don’t get COVID.

  • Kill Old Contracts Before They Kill You: MPLS and OC3 weren’t just expensive—they were inertia. Shedding them accelerated everything else.

  • Inventory Is a Verb: Recon first. Know what you own, what owns you, and what needs a Viking funeral.

  • Hotel Labs Are Legit: If you can’t get into the building at noon, make your own building at 9 p.m. Repeat until done.

  • Hardware Is a Suggestion: In the absence of servers, a well-spec’d laptop with good I/O and disciplined virtualization can keep a company alive. Would I run like that forever? Absolutely not. Will I do it again in a pinch? Without blinking.

  • Security Theater Is Expensive: Don’t pay ransoms until a forensic adult declares the data actually encrypted.


The Human Layer (Why This Worked)

Every site was its own fiefdom—habits, heroes, holy relics. Technology was half the battle; the other half was anthropology. We had to earn permission to untangle the spaghetti. That meant telling the truth, sharing the plan, and being unoffendable.


Most teams have a dropout rate when the fire gets hot. Ours didn’t. We disagreed loudly and worked harder. No passengers. Everyone drove. That kind of crew turns impossible into Tuesday.


We ate gas-station jerky when restaurants were “maybe open.” We drove through ice and dodged horses with better lane discipline than humans.

We installed at midnight and explained ourselves to cops. We staged racks in hotel rooms. We took our victories in Apple stickers and brisket fries and spice jars from kind strangers.


This wasn’t just an IT project. It was a road novel with patch panels.


Field Notes You Can Steal

  • Design for distance. Assume your people, phones, and files will live in different zip codes. Make that boring.

  • Shrink the blast radius. Standardize stacks. Kill snowflakes. Snowflakes are pretty until they avalanche.

  • Write the runbook you’ll need at 2 a.m. If the only copy of tribal knowledge is in someone’s head, you don’t have a plan—you have a liability.

  • Treat “legacy” like hazardous waste. Label it. Contain it. Migrate it. Don’t cuddle it.

  • Measure reality, not reputation. That “hot” server room was a freezer with dusty lungs. Trust sensors, not stories.

  • Encrypt like you mean it. And when attackers don’t, don’t fund their incompetence. Verify first.



6th day in hotel they got real food!

Epilogue: Why I’d Do It Again

Years later, that trip still hums in my bones. It was a perfect storm: inherited entropy, pandemic logistics, conflicting politics, and a small team with more stubbornness than sense. We did not have the luxury of perfect. We had the obligation of done.


And here’s the truth I carry into every boardroom, plant floor, and classroom since:

  • Technology is for people. If they can’t use it, you didn’t finish.

  • Clarity is a force multiplier. Put the map on the table. Invite critique. Ship anyway.

  • Improvisation isn’t recklessness. It’s disciplined creativity under constraints.

  • The right crew beats the right tool. The tool shows up in a box. The crew shows up when it’s snowing.


If you’ve read this far, you’ve ridden in the Jeep. You’ve felt the oven heat, the lake wind, the server dust, the bourbon glow. You’ve seen a Windows estate run on a MacBook and a ransom note turn into a undo script. You’ve lived the punchline: sometimes the fix that “shouldn’t work” is the only one that does—and that’s okay, as long as you land the plane.



Comments


Animated coffee.gif
cup2 trans.fw.png

© 2018 Rich Washburn

bottom of page