Preface: I work in telecom. However, by now I had left [Telco] and began cross training at a [Data Center]. This is a story from those (literally) dark times.

$BT - Me

$DSL - Day shift leader

$C1 - Customer 1

We begin this tale, with me having left [Telco] to work at a [Data Center] for more money. While I had IT certifications at this point, the [Data Center] gave me the chance to actually put them to use, while also keeping my telco skills alive.

Of course, more money made the decision quite a bit easier to make.

I'm a chipper person.

No.

Seriously.

But I feel like we've had this conversation before.

The exception to this is when I have to work night shift.

Night shift $BT feels a bit like a photographic negative of the day shift version. Whereas morning $BT needs three cups of coffee to calm the fuck down, night shift $BT needs the same quantity of Fair Trade organic brew just to hit awake and semi-functional.

Side note:

I don't actually give a fuck about Fair Trade. Years of idiotic behavior on the part of coworkers and customers has made the tears of child labor taste all the sweeter.

Kidding. Kidding.

So it was with Night of the Living Dead speed that I strolled through the door(s) of the [Data Center] one fateful Thursday night.

$DSL - Ah, $BT, you're right on time! You're flying solo tonight.

$BT - ...

$DSL - Your partners in crime are out for the night, so you'll have to man operations on your own.

$BT - ...

$DSL - Should be a quiet night.

$BT - ...

$DSL - Let's get started with the hand off.

Side note 2:

The [Data Center] I worked in ran in 12.5 hour shifts with a half hour overlap between them. This was designed to allow each shift to inform the other of the day’s/night’s events.

$DSL – Alright everybody, all of the cross connects for the next three days have been completed ahead of schedule. The remote hands work has been done. And we actually managed to get all of the deinstall orders done as well.

$Internal BT - Oh no.

Mandatory side note 3:

[Data Center] was carrier neutral. This meant that from high speed traders to media companies, everyone was treated equally. It didn’t matter who you were, if you wanted 400 square feet of space with two 40 amp feeds, you paid the same as everyone else. Think of it as a large, brick shit-house of a building where we owned the power, cooling, space, and the interconnect (aptly known as a cross connect) between customers.

The reason for this existing is imagine you have [Internet Provider 1’s] service. I have [Internet Provider 2’s] service. Those networks are physically separate. However, you have a website selling band merch that I want to buy. Somehow my data traffic has to get from my internet provider’s physical network, to yours.

So what do we do?

Your internet service provider isn’t going to put their edge routers (the [usually] literal edge of their network) in my provider’s building and the same goes for my internet service provider.

Enter a neutral party.

They provide everything needed (including security) to maintain equipment and link them together, without either of the (non-neutral) companies being liable for the other’s equipment and traffic.

Cue back to story.

Deinstalls were the bane of our existence. I can’t tell you how many times someone had asked for a fiber optic line to be deinstalled, only for another part of the same company to flip their shit when we actually disconnected it. Of course, companies pay for cross connects, so the longer they stay up, the more we bill them (at this point unnecessarily).

So pull it down and get bitched at by the left hand or leave it up and get bitched at by the right.

It was a stupid catch-22.

To counter this, we had protocols in place. We would disconnect the line from their panel and tag it, make notes on it, and then wait 72 hours. If after 72 hours you weren’t bitching about your OC-192 (10-ish gigabits per second) circuit being down, it was pretty safe to say you weren’t going to. If they did flip about it, we simply plugged it back in, got the information of the person doing the bitching, and then forwarded everything back to the first person who asked for the line to be disconnected. In essence, we let them fight it out amongst themselves.

CYA was our definitive motto.

All of those rules, however, went out the window for non-payment.

When a customer refused to pay their bill, we tried to work with them. Generally, large companies tend to run behind on their bills, so we weren’t aggressive about disconnecting for it. But after a year of non-payment, multiple attempts at negotiations, and official legal notices being involved we would reclaim our space. This usually involved systematically disconnecting each and every single one of their cross connects, sending notifications to everyone on the other side of those cross connects, pulling power from their equipment, and then palletizing all of their gear (routers, switches, customer provided panels, etc.) for shipment.

Typically this would happen over several weeks in order to avoid a flood of calls and trouble tickets.

But we had this one [Customer]…

$BT – You finished all of the deinstalls in the queue?

$DSL – Yep.

$BT – Even the cage for [Customer]?

Side note 4:

The customers at the [Data Center] literally had mesh cages separating each other. Each cage had its own number.

$DSL – [Executive Director] said he wanted everything out today. They haven’t paid us for two years and he said it had to be done ASAP. We even called in the swing shift guys to help out. And got the engineers to pull power from everything so we could get their equipment unplugged and palletized.

$Internal BT – Fuck. We’re out of K-cups.

$BT – Which cage was it you guys disconnected?

Shuffling through notes.

$DSL – 905.

$BT – Sounds good.

Then out the door they ran.

I can’t blame them. Honestly, after 12 hours in a windowless building most people want to leave. Not even the, “supplemental solar,” lights that our management had installed were enough to make up for the lack of good old UV.

$DSL wasn’t lying when he said that they had cleared out our ticketing queue. Barring the few packages that still needed to be picked up by their owners, there really wasn’t much to do that night.

It was with great surprise then, that 3:30 AM rolled around with a very, very agitated looking Asian man knocking on my operations room door.

$BT – Yes, sir. How can I help you this morning?

$C1 – I have a trouble ticket for our cage, and I need to see if my package is here.

$BT – No problem, sir. What’s the [tracking number] for the package or the ticket number for our system.

He provides it. I look it up.

$BT – It says here that it was delivered to your cage yesterday.

$C1 – Oh, okay. Sounds good.

Sweet. I love fixing simple problems.

Thirty minutes go by. And he’s back standing at my door.

$BT – Yes, sir. How can I be of-

$C1 – What the fuck did you guys do!?

To be continued.

Edit: Part 2 is up. If anyone cares to read it, feel free.

Edit 2: Part 3 is up as well.

Edit 3: Gold!? I'm honored. Thank-you gilding user (I've replied to your note). I'm glad you all love my stories.

I have plenty more to share.