This post has been de-listed
It is no longer included in search results and normal feeds (front page, hot posts, subreddit posts, etc). It remains visible only via the author's post history.
This is a little long, sorry. Once upon a time I was a part-time PFY for the NOC of a large urban university....
Dramatis Personae
JC - me
PFY2 - another part-timer
PB - Pork Bun, a full-time staffer who would occasionally bring delicious roast pork buns to work
SW - Scalliwag, another full-time staffer.
M - our mostly pretty decent manager
It was mid-afternoon on an otherwise normal day, when our monitoring software started throwing up a bunch of red and yellow items on the screen. Not just one or two, either, but a dozen different blinking items. Checking over the switch and router names, it looks like ... yep, every piece of equipment in four buildings is unreachable.
I tried some more Cisco-fu on the malfunctioning equipment, with no luck. Checking our documentation, it turns out that all of these buildings uplink through the same switchport in a nearby building. Score! I logged in to the uplink only to find a null signal on that port ... and nothing that I can do will change it. Time to bring the problem to one of the full-timers - they have more experience and higher level access.
JC: Hey, PorkBun! Buildings W, X, Y, and Z are all down. They all uplink through Building V, but I can't get anywhere with the uplink port.
PB: Hmm, okay. [minutes later] Yeah, it's not responding to me, either. Grab a tool kit and let's head out there.
After a brisk walk and about half an hour on-site in the basement of Building V, nothing has worked. Power cycling the equipment, plugging in a laptop to the management port on the switches ... nothing. Almost on a whim, PB decides to check the fiber that links Building V to the other four, mostly to make sure that the fiber end hasn't somehow become damaged.
PB: Well, the fiber looks fine. ... wait ... wait, that's not ... it says that the fiber is only about 100 meters long. It ought to be much longer than that.
Unfortunately, there's no documentation on where this fiber was run, exactly, so we have to follow it by hand as best we can, through conduit and over pipes, to a security gate. A gate that, unlike most doors in these basements, we don't have a key for. We call Facilities and politely ask for help. ...eventually, a maintenance guy stops by and unlocks the gate for us, but won't let us borrow the key. We prop it open with a stray cinderblock for the time-being and continue on our quest.
...and it's there! The fiber was up on the side of the building, where some mysterious bastard had cut it. Why they coiled it up nicely and wrapped it around a nearby pipe, I'll never know. ...I'll also never know why someone ran the fiber across a back alley to begin with, but that's another problem.
At this point we've been out here for two hours, and it's well after 5. PorkBun calls Manager, to fill him in on the details and see what Manager has to say.
Manager fortunately lives nearby and decides to just drop in on us, along with PFY2. After some thinking, an arrangement is reached. PorkBun is going home and SW will be summoned - because SW is the only staffer available that evening who knows how to use the fiber splicer. PFY2 and I will hang out on-site until SW shows up, because we can't close that gate, and can't leave it propped open without a guard. Since it's going to be at least an hour before SW makes his way in from the suburbs, Manager offers us overtime and drops off a pizza later. Like I said, Manager was a pretty good boss.
SW finally appears, armed with the fiber splicer. If you're not familiar with the process of splicing optical fiber, then ... well, it's an experience. You have to carefully cleave each end of the fiber to be flat, clean off any stray dust, and then very carefully align them in the splicer. Once you think that you have the fiber ends lined up with each other, the splicer moves them together and melts the joint to connect the ends. If your alignment isn't just right, however, then it's back to step 1. Each attempt takes at least half an hour, since SW doesn't do this very often.
Since we're in a back alley, there's really nowhere suitable to set the splicer and work. SW sends PFY2 and me into the other basement rooms to find a good work surface. Eventually, we locate a platform stepladder in the boiler room, and bring that out into the alley. SW sets up the splicer on the platform, and ... after about five very long tries, one of the fibers is good to go!
At this point it's after midnight, we're all worn out, and there's no way in hell that we're going to do the rest of the fibers in that cable. One fiber will give Buildings W-Z some connectivity, and that should hold them over until the field crew can replace the fiber entirely. We start to break things down.
And then we realize that something is horribly wrong. In his haste, SW had run the fiber through the legs of the stepladder before splicing it. The three of us stop and stare at it for a moment.
SW: I am NOT splicing that again.
JC: (joking) There's a hacksaw in the tool bag, you know.
I went to throw out some of the trash from the splice - bits of fiber, alcohol wipes, etc. When I returned, PFY2 was holding the ladder while SW cut through the legs with the hacksaw.
SW: There's a dumpster around the corner. Make it disappear.
...and nobody saw the ladder again.
tldr: Mysterious assailant possibly armed with a meat cleaver, overtime, melting things, hacksaw, getting rid of the evidence.
Subreddit
Post Details
- Posted
- 10 years ago
- Reddit URL
- View post on reddit.com
- External URL
- reddit.com/r/talesfromte...