Preface: I work in telecom. I've worked at a few different companies, while climbing the ladder, but this story is one from the early days of my career, at a company that I affectionately call [Telco].

$BT – Me.

$NOC – NOC agent with [Telco].

$MAN – My frontline manager.

$2L – Second level manager. Drill Sergeant in another life, supposedly.

Organization is something close to my heart.

I thoroughly enjoy when people (clearly) document, organize, and maintain records.

It’s one of life’s greatest pleasures.

As you can imagine, not doing so can lead to some serious problems.

Imagine my surprise when, after several months with [Telco], things seemed to be in good working order. I had found that as long as I did my part, the system did its part.

For instance, let’s say I received a trouble ticket in [Dispatch System]. I went out, fixed the problem, documented what I found, and that was then updated to our database. Then, if another issue arose with the same customer, I could easily look back and see the previous issues that had arisen. In fact, I could look at a customer, cable pair, or really any item that had a database assignment, and find affiliated tickets.

It was a lazy Sunday afternoon, when my blissful reality came to a screeching halt.

I was halfway through a ham and vegetable omelet (don’t judge, I love eggs at all hours), when my company flip phone went off with a 911 alert.

Then another.

Then another.

And it kept going.

Next thing I knew, I was staring at forty [Dispatch System] tickets (and counting).

I try to be logical about things, but the fact that my phone was blowing up had sent me into full panic mode, so I began looking through the tickets on my company laptop, hoping to make sense of what I had found.

It was weird.

I needed more coffee.

There were tickets ranging from as far back as two months prior to that day, up until that very morning. We had downed fiber circuits, copper circuits, cell towers, and more. There were business circuits that were taking trickles of errors, power tickets, and configuration mismatches. It was incredible the amount of things that were sitting there waiting to be worked.

At first, I figured it had to be some mistake, that somehow old tickets that had already been worked were duplicated and dropped back into my queue. So I did what any good technician does, and I began to dig deeper.

$NOC – This is [Name] with [Telco] NOC, how can I assist you?

$BT – Hi this is $BT, I’m a technician with [Telco] in [City]. I have quite a few tickets that have shown up, and I was wondering if you could assist me with them?

$NOC – Sure, what are the numbers?

As I ran through the list, I could hear him mumble under his breath about the sheer quantity of them.

$NOC – Well, as far as I can tell, everyone single one you’ve given me so far has been valid.

$BT – What do you mean? You mean all of these issues are currently outstanding?

$NOC – Well yeah. We have a dozen or so cases of T lines being down, four fiber outages, a bunch of individual circuits taking errors, and quite a few smaller cases of trouble on VOIP systems.

$BT – I get that, but are you sure none of these are duplicates from prior dispatches?

$NOC – I’m going through and logging into the systems these circuits run through, and so far, every one that I’ve gone into that is showing as, “down,” is still on protect. I’m betting these were just trapped in the system and someone finally released them.

$BT – But some of this stuff is two months old.

$NOC – I don’t know what to tell you, bud. Maybe you just caught a glitch.

Eye twitch.

It was a Sunday.

Even if I worked the next 24 hours straight, there was no way I was going to get every single ticket fixed. I assumed that some of the issues were related (a single cut fiber could cause multiple smaller copper lines that it feeds to go down), but that still left more than I could chew.

As I hung up with the NOC and began prioritizing my workload, my boss called.

$MAN - $BT! What’s going on out there?

$BT – I have no idea, I just got off the phone with the NOC.

$MAN – I assume you’re going to go take care of this?

$BT – I’ll take care of what I can. I was just trying to prioritize everything so that I could clear the big stuff off our plate first.

$2L – Just get it done.

Great. My second level was on. I’m glad I hadn’t said anything that would cause him to hate me more than he already did.

Side note:

My second level was a total asshat. Seriously, if he’s on here and knows who I am, he can suck it. I have never met a bigger egomaniac who knew nothing about their job than him.

End rant.

$BT – Well, there are over forty tickets here, and I can’t be in two places at-

$2L – I don’t care, just get it done. I’ve got the VP of operations breathing down my neck, wanting to know why we have so many trouble tickets in our queue.

$BT – Well sir, if you have the VP’s ear, maybe you can convince him to help us find out why these were all dropped at once.

-Click-

Charming fellow.

$MAN – Looks like he dropped off the call.

$BT – Yeah, can you talk to [Other Tech] and see if he'll help out with some of this?

[Other Tech] was the technician in the territory next to mine. He and I had weekly, “meetings,” (i.e. lunch on the company dime) to talk shop and to shoot the shit.

$MAN – Sure, I’ll give him a call, as well as some of the others to see if they can work the stuff closest to their areas.

$BT – Thanks. I really appreciate this.

-Click-

After a few hours, I had managed to solve some of the issues.

One of the fiber cuts was a mouse chew not far from where I lived, so after a couple OTDR shots and a phone call, I had my normal splice crew out there fixing the fibers and sealing up the manhole to make sure nothing could crawl back in.

The other fiber cut was a tree that had fallen and was leaning on our line. Not a, “cut,” per se, but the fibers were still broken. A quick trim and a phone call later and I had a decent contract crew out there fixing it.

That killed off two of my fiber cuts and a dozen or so affiliated tickets related to it.

By now, I had worked up a spreadsheet based on the ticket information and our database, to map out what tickets were directly related and what could be worked that day.

Obviously, many businesses weren’t open on the weekend (especially not on Sundays) so anything related to customer premise fixes had to be shelved until Monday.

Anything in the backbone or at the cell towers could be worked immediately.

And any power tickets related to those could either be kicked back to the cell tower owner or moved over to our power technicians (we had dedicated techs that only did power work).

In short, I was feeling pretty good about my chances. More than half of the severe issues were solved, and I still had a few hours left in the day to solve what remained (of what could be worked that day). Once Monday hit, we would have a full crew of guys that could tackle the rest.

As I went through my spreadsheet to double check my work, my phone rang again.

It was my boss.

$MAN – Good news, $BT.

$BT – Okay.

$MAN – I couldn’t get [Other Tech] to help out. He’s out of town with his kids this weekend.

$BT – So what’s the good news?

$MAN – [Tech 3] has offered to help.

$Internal BT – Fuck.

$BT – Great.

I was screwed.

To be continued…

Part 2 is up!