We’re replacing our 20-plus-year-old NEC PABX with a VOIP phone system. While it’s a big learning curve for me it’s coming together and we are on track for go-live next week.
$offsider and I went to roll out the remaining IP phones this morning, the comms room for which also serves the room we will be using for training on the new phone system. So while $offsider was putting phones on desks I quickly powered up the spare Power-over-Ethernet (PoE) switch (that the training phones would connect to) and plugged it into the network.
Within a couple of minutes $offisider is in the room: “Did you take down his connection already?”
“No…” I said, as someone from the call centre came barreling in to tell me that the whole team had just lost all network connectivity.
“Well, it must have been this switch…” I said, as I was interrupted by a call from $boss. “What did you do?” he thundered. “The network is down EVERYWHERE.”
“Well, I just plugged in this switch… and I’ve unplugged it now,” I said.
“OK. Still isn’t up,” he replied.
Oh bugger.
I ran back to the server room and noticed that the two switches in there were behaving strangely, so I power-cycled them… to no avail. Realising that the problem was most likely where I’d just been, I legged it back to the other building and called $offsider, who had gone back to the office to man the phones.
I got him to set up a continuous ping to a DC, and pulled the fibre uplink on one of the switches. Immediately $offsider reports a response. Resetting the switch elicits the same response but the ping is dropped once the switch comes back up.
Hmmm, methinks this switch is hosed. Plenty of spare ports on the other switches in the rack, so I re-patch them all. Suddenly $offsider tells me the ping has gone down again; some more troubleshooting reveals that the offending port is actually an uplink to a third switch. So I start repatching cables from that switch into the one previously thought dodgy. One patch lead in particular seemed to be causing the problem, so I set about tracing it back to… the same room that I’d plugged in the training switch 45 minutes before.
Sitting innocently right next to the now-standalone Cisco PoE training switch was a D-link ADSL router that had been connected to a network port under the meeting table and had been serving as a hub for a couple of years. Unplugging this from the network restored order to a chaotic morning.
The D-link has now been filed under B.
TL; DR: A “rogue” hub causes pandemonium after a new Cisco switch is connected.
You could even get them yellow shirts or something.
Ops wore red, just like security. 
Yellow’s for command.
