Two from the old job:
The Half Life Server
Back in 98 to 99-ish, we bought a lot of Compaq towers. Like, I (as the Asset Guy/Level 1) would get a skid of PCs in every month, sometimes two! We bought them and had minimal support, so after a year or so would have a lot of spare machines for parts and such.
And so I moved to being Phone Guy. At around that time, we were playing a lot of Half Life in the office. Management was cool with it as long as it was after hours and That One Guy kept his voice down.
So I still had access to The Room, and built a server. Old Compaq tower, missing a few parts, running linux headless. It lived in the asset room as most of the senior server admins weren’t gamers (OK, one was big into Ultima Online, but that’s different) and we couldn’t sneak it in to a space that was overflowing. Still, it lived in the storage room/workshop and allw as good.
Until I went on vacation, and some tech stole the #$^$&*@ hard drive to try and fix another machine. 
Voicemail and Horse Trading
In time the company needed to replace the OS/2 based voice mail system. We planned to get what eventually became Cisco Unity, but due to it being bought by Cisco and another office not wanting us to have shiny toys no one else did (unified messaging) we ended up going with the shiny new package from Telco manufacturer that offered as a key feature, Unified Messaging. I didn’t say this story would make sense, did I?
(Aside: Said package was a piece of crap I regret to this day. The developers were a mix of either:
- Deep Hardware guys, sued to programming bare-metal PBX systems.
- Brand new developers using Delphi that might have been competent if they were doing an application, not a complex server with lots of real-time bits and hardware integration.
The Voice mail package was a piece of junk. Hardware was limited to a single-processor, and it ran as a bunch of applications you needed to leave open. Instead of proper Windows Services, the VM processing was an application, as was backup, administration, an app to run the DSPs, etc. We went through versions like crazy for a while as we were early adopters. The unified messaging was never rolled out as it sucked. It used an open-permissions share on the server that seemed very exploitable, and due to not playing ball with Microsoft every PC required a special app that would hit a ‘Yes’ button for them when a dialog about integration popped up…)
Back to the server. Due to Reasons the Company bought the server themselves. We had big corporate deals, wanted to be uniform, etc.
And that’s where it went wrong, leading to a Frankenserver.
Server comes in, vendor starts looking at it. Voice card won’t fit. At that time, for voice stuff you’d end up with what resembled a big multi-line network card that had the voice ports (sometimes RJ-45s that would get split out to individual phone lines) and DSPs ti handle the voice processing. These tended to be somewhat big. This was a full-sized card.
Small but perfectly formed server was a 1u or 2u that could only accommodate half-height cards. This presented a problem.
For Reasons return was not a good option. I don’t remember why: Vendor didn’t want to admit they messed up (they had vetted the new server before it was ordered), project was already way behind, etc.
So we went trading. Ended up getting “the Beast”: a retired Dell server from the Server Team that was impressively large: multiple drives, dual processors, hot-swap PCI, something like 8 gigs of ram. (This was in the early 2000s when 8 gigs of ram was a big deal.) Pretty sweet box, if a bit outdated. It was a beast. I’m thinking something like 5u, maybe 6?
And then it got ugly: Had to get a working OS for this beast (predictably, not easy) and load the voice mail applications. They crash every time we try to start them. Vendor goes to manufacturer. Problem is that their app can’t handle the dual processor goodness. Maybe next year’s release.
So we pull a processor. Which causes the box not to boot. Turns out this big Dell was such that you needed to put a blank card in a processor died or it wouldn’t boot. Vendor ordered that, more delays, etc. etc.
Finally got it up, and it ran for several years. It was always a problem. Restarts were painfully slow (due to mem-testing 8 gigs) and the software would crash if you looked at it wrong. Had to keep a small partition, even though it had a RAID and tons of space.
We finally replaced it with another voice mail package. This one was kind of scary, too: First, it was essentially Cisco Unity form an alternate dimensions. An offshoot with some of the same developers after they’d been bought and spun off by the Big C. Still strangely similar paradigms, which were better than the mess that ran before. This one was an ‘in-skin’ system, which meant they got it running on a Geode processor and a laptop hard drive and wired it onto a PBX line card. Still better than the Beast it replaced.
I left the Beast in place. It might still be there, actually. At first this was to just have it in case Vital Business Data was identified as being left behind in the migration, then it was because pulling the old cabling was kind of scary. Then I just stopped caring.
