Recap: A whole lot of tickets showed up out of nowhere and some unexpected help came my way.

Part 1 for those wishing to read it (and know what’s going on).

$BT – Me.

$MAN – My first level manager.

$T3 – The aforementioned [Tech 3]. Bringer of chaos. Lord of disaster.

When we last left off, I was in the middle of cleaning up a giant mess when my boss alerted me to [Tech 3’s] desire to assist.

$BT – Boss, he’s less experienced on that equipment than required. Are you sure you want him working on it?

I had to be careful. Despite all of the things [Tech 3] had done up to this point, I wasn’t going to disparage him in front of my boss and risk not looking like a, “team player,” while a major incident was going on.

$MAN – He’ll be fine. He’s gone to the class for it.

Great. A class. I’m sure he’ll just be a fount of useful knowledge.

$BT – Alright. I’ll check in on him in a few hours for an update.

$MAN – Sounds good. Take care $BT.

-Click-

I was screwed.

I knew I was screwed.

One of the pieces of equipment that was having issues was a [Brand] multiplexer. On one side of the multiplexer we had a fiber, “feeding,” it, and on the other side were a set of cards that made up a series of smaller circuits going to various customers (in this case DS3 circuits). For some reason, the remote access and monitoring channels were down, so a technician was needed to verify that the chassis was functional and reset the configuration to bring back remote access.

The work [Tech 3] needed to do was simple:

The technician would need to log into the chassis from their laptop, back up the existing configuration (if it hadn’t been dumped), build a new configuration from our in-house, automated configuration builder, then load it into the chassis as a sort of, “fresh,” install.

Building a new configuration was usually simple if you knew what you were doing. The entire process was automated. It was very difficult to mess up if you read the fields you were entering.

Side note:

Seriously, I could train a child to do it. We had spent months testing it before it went live. It wasn’t foolproof, but considering you weren’t supposed to be a fool and working in the field, that requirement wasn’t viewed as applicable.

What you didn’t want to do was manually edit the configuration file.

Manually editing the file left too much room for error. The file type had to be simple text. It couldn’t have excess returns in it. There were rules regarding syntax, and the spelling of everything had to be exact. The system was extremely picky, and on more than one occasion technicians had tried to shortcut and edit things by hand (to avoid having to use the time consuming configuration builder), then ended up loading a config that locked out the chassis.

[Tech 3] had gone to the school for this work.

[Tech 3] should not have had any problems using our config builder.

After a few hours had passed, I managed to corner my third fiber cut. It was located in a riser where the fiber had transitioned from underground to aerial (and vice versa, technically) and was a known bad spot (another story for another time, all hail Mighty Mouse). Once my in-house splice team was on the scene (they had buttoned up the first cut by this point), I knew it was time to check in with my comrade in arms.

-Ring-

-Ring-

-Ring-

“Hi, you’ve reached [Tech 3] with [Telco]-“

-Click-

Not happening.

I waited another thirty minutes before trying to call him again, using the time to check back over my spreadsheet.

Voicemail again.

The site where he was located was twenty minutes from my house. I didn’t like stepping on another tech’s toes when they were working, but as he had a history of causing problems and was doing work in my area, I felt it was okay this one time to stop by for a friendly visit.

As I drove through the small town where I lived, a slight feeling of dread began to creep up the back of my skull. I knew that [Tech 3] had done something. This was an hour long job tops. Plug in, back up, build new config, load, and then call the NOC to verify they could access it.

There was no damn way he should still be on site.

And he was.

There, three hours after my boss had spoken with me was [Tech 3’s] truck sitting in the parking lot.

Fuck.

I walked through the doors, hoping that he was just hiding in the bathroom to avoid work and soak up overtime.

Unfortunately, he wasn’t.

$T3 – What do you mean I don’t have to copy it? How else am I supposed to do it?

He was on the phone with someone and clearly pissed off about being told what (not) to do.

$T3 – Of course I did. I went to class for this the same as you.

Oh, great argument.

He spotted me, and then hung up the phone in a hurry.

$T3 – Hey, $BT! How’s it going?

$BT – Good. I’m just stopping in to see how things are going over here.

$T3 – They’re good. I’ve run into a small wrinkle with the config builder, so I had to call [Ops Support Tech] to straighten it out.

Side note 2:

[Ops Support Tech] was one of the in-house personnel who helped write the configuration builder.

$BT – Oh? And what did he say?

$T3 – He said that I shouldn’t be copying the information.

Wait. What?

$BT – Show me the error you’re encountering.

I watched as [Tech 3] opened up the configuration builder. Good.

He inputted the information into the program. Good.

He verified all of the required information was correct. Good.

Then he proceeded to highlight everything on the screen with his mouse, and copy-paste it into a .docx file.

What the fuck.

$BT – Um…you didn’t write the configuration.

$T3 – Yes, I did. Did you not see me copy it?

Hang on a second.

$BT – Did you already upload a file to the chassis?

$T3 – I tried. But it kept giving me an error.

$BT – What error?

$T3 – It said the file type was wrong.

$BT – Move. Please.

I took over.

$BT – What type of file is supposed to be loaded into the chassis?

$T3 – A text file.

I pulled up the properties of the file he was copy-pasting to.

$BT – And what does that say?

$T3 – It’s a Word file.

$BT – Yes, but it’s a .docx. The chassis can’t read a .docx. It doesn’t recognize a .docx because a .docx is not a .txt file!

I was pissed. My coffee cup was empty.

$BT – Look.

I again verified the information he had loaded into the configuration builder and then hit the bright green, “Build,” button at the top of the page. As soon as I did, a prompt came up asking me where I wanted to save the .txt file it was creating. I made sure the name was simple and then loaded it to the chassis.

$BT – Call the NOC, verify it’s up, and then go home.

Wonderment filled [Tech 3’s] face.

$T3 – How did you do that?

Epilogue: Fortunately for me, the chassis came back up. Somehow, the existing configuration had been corrupted (we were only able to determine that the issue was more than a month old), which left the chassis up and running, but the remote management down. It wasn’t discovered until a customer called in about a problem (packet loss) and the NOC couldn’t remote in to check.

[Tech 3] got a tongue lashing from our second level for wasting overtime, but got away relatively unscathed.

A few weeks after the whole incident happened, it was discovered that the cause was a dispatcher who had been recently fired. Throughout her day, when she couldn’t figure out where tickets should have gone (apparently a common thing for her) she placed a hold on them. When her access was revoked, all of those tickets dumped into the system and were spread out across two dozen or so other dispatchers, so no one realized how severe the situation was at first. Apparently, my tickets were just a small fraction let loose from the year or so she had been there (I had only been with the company for a few months when this went down).

After that day my workload increased quite a bit, as the full scope of our network issues was unleashed upon me (and not held up in some dispatcher's queue).