Oh, it’s been a while. Where to start? I’ve been at my current job for nearly three years. Here’s something from the first year…
So, I start getting hammered with tickets from all the small sites. It’s unusual for a Friday. Then I got tickets for various applications timing out. One of my needier small sites starts piling on tickets. They know they have an open ticket, but the staff feels the need for each person to call in a new one.
It doesn’t take long to determine the network is down. Not just for $needy_site, but for many sites. And my phone is amazingly silent. There are a few folks with my direct number, and I can’t believe they wouldn’t call me to bypass the Help Desk for something as big as a site outage. These people call me when the fax machine jams.
That’s when I notice the phone support folks are darting around the building, too. I make some calls, and find out there’s no phone service to many sites. Only internal calls are working.
I stop $phone_support_guy. He says, “Yeah, we know. The phone lines are down all over the place, too. No one knows why yet. We’re getting the minimum services running through… alternate means.” Translation: We’re down to stretching string so the paper cups will have better coverage.
The network is a mess. Server traffic is also being redirected, and some of the apps just can’t handle the strain. A few apps are holding up, but users are hammering those programs and they’re getting slower by the minute.
I grab someone in the higher levels of network management and demand answers. That’s when I learn the horrible truth.
Some idiot decided that since the weather has been sunny, it was a great time to redo his entire backyard. It’s also been very hot, with record-setting temperatures, so $idiot decided it was not a good idea to spend the whole day digging. No, $idiot thinks, it’ll be much faster to borrow a backhoe and go crazy with it.
Y’know those billboards and brochures from the utility companies? They say, “Call before you dig!” They’re not kidding.
$Idiot took out phone and network trunk lines for half of the city, starting around 10am. We’re not the only ones who are screaming at the ISP and telecom companies. All that can be done is to wait for repairs to the massive underground cables, and to get users to lay off the secondary connections so that essentials will continue to work.
$Idiot should be thanking his lucky stars that the ISP is not releasing the address of this bunglefsck. There are a few engineers who’d like to show him the importance of backhoe safety – from the ground’s perspective.
I am very sad for the people who are on-call this weekend.
TL, DR: Call before you dig. Seriously.