Right after 9/11 happened, I had to go from Chicago to our New York City datacenter. We're a stock exchange/broker, and this datacenter was 2 blocks from ground zero. There was talk that NASDAQ wouldn't be up and we would have to take over orders for them interim. While we had 6000 servers in Chicago, we really needed to get our 4500 in NYC up too.

So we're hustling and taking shortcuts when we could to try to get things up and running. Cooling tower drained, cleaned, refilled. A big ass semi-trailer sized generator dropped off. Begin to fire everything up. We're looking good.

Now it's time to start the backup system/power smoothing and switch from mains to it.

Now this power smoothing system was an ancient beast. We inherited from the previous owners - which was a 'big iron' mainframe setup from the 60s. Now it smoothed power by using a flywheel. I don't know how it all worked, and it had never needed to be restarted before, so this was all new to me, a young 24 year old sysadmin. The flywheel was a 10 foot high solid concrete wheel that weighed 4 tons. It spun at 5000rpms. To start it, you took off an access door, applied power to the system, and gave it a little 'nudge'. There was an arrow in chalk pointing one way, so I 'nudged' it in that direction and it began spinning up. After 2 hours, when it was fully up to speed, you took a timing light and shined it at it to make sure it was going the right way - yes, the system could run in either direction. So, all looks well. So I grab this huge lever that looks like it's right out of a Shelley novel and slowly push it from 'BYPASS' to 'ACTIVE'. As soon as the lever went past the "T" in 'ACTIVE' we heard a dozen large BOOMS, and then a bunch of smaller 'booms'. Complete darkness.

Apparently at some point in time the wheel needed maintenance. And when they were done, they put it on backwards. So the arrow was pointing in the wrong direction. So instead of positive voltage, I was sending thousands of volts of negative voltage to all of the PDUs and the battery racks. As you could imagine, they don't expect that kind of thing. The big booms were the $50,000 power distribution units exploding. The smaller ones were all of the control circuitry for the UPS.

And thats not even the worst bit.

So all these booms happen. Then it's dead quiet. We managed to destroy this semi-trailer sized rental generator. But remember how I said it was a 4 ton flywheel? And it took 2 hours to come up to speed? It took roughly that long for it to spin down, as well. And I didn't flip the lever back to 'BYPASS', so this system is still pumping voltage. 6 fires later....

It cost the company close to 2.5 million dollars to get the system working again - and it took over 3 weeks of around the clock work. We're talking replacing everything - the PDUs, the switching system, even the wires down to the street.

Didn't get fired though. Damn I miss that place.

Edit: source