I’ve not had time to post the past couple of days due to a bit of a crisis at work . .  our Office suffered a number of power surges on Monday that resulted in our Backup Domain Servers power supply blowing and every single switch in our patch cabinet doing the same, this all occurred around 12 noon, due to this we had to start shutting all our server’s down, thankfully most of them had a UPS attached so gave us a bit of leway to get it done before we had to shut off all the comms room power.

Once we’d recovered from the initial shock of it all we were able to discover that a Power Panel in the comms room had been arking because the grub-screw holding the neutral in place had worked lose, we soon got this sorted and that’s when we discovered the switches had blown, by this time it was getting on for 2pm, now trying to find 5 x 24 port switch’s at 2pm is not an easy job, between me and Tony we must have rung just about every comms supply company in Manchester, we eventually managed to get some from a support company we deal with in Liverpool, they got one of there guys who lives in Manchester to bring them across with him on his train journey home, but due to delays on the train we didn’t actually get them till 7:40pm, in the intervening time me and Tony started to strip the patch panel to give us clear access to the switch’s, it was a fcuking mess I can tell you, couldn’t get to the switchs because everything was so tight in the cabinet

The Panel 1/2 through being stripped

While Tony finished up stripping the panel I went to get the switchs from the train station, inadvertantly forgetting my wallet, meaning I had to borrow £2 off the guy who brought them to us just so I could get out of the NCP Car park (nightmare averted).

After relabeling the patch cables and installing the switchs the time was approaching 11:00pm, by the time we got to actually testing everything it was near enough midnight, we eventually left the building around 00:15am the following morning, this was nearly 7 hours after we should have normally been leaving work, but at least we thought nothing else could go wrong. . . .

Tuesday morning got to work to find everyone at work locked out of the building, it now turned out that the power surge hadn’t just knocked out the switch’s it had also knocked out our buildings access system, which meant every single maglock enabled door was locked shut, so we had to force our way through 3 of these doors just to get into the building, an engineer was called to sort this problem, once he’d been contacted me and Tony got back to checking the system properly for any more problems, which there were many of, seems a jetstream had also blown resulting in 1/2 the warehouse being off the system, we got this resolved only to discover that the phone system had also been  effected, again an outside engineer was called to resolve this, we also discovered that our ActFax server and our EDI server had also been effected, thankfully only having there external modems blow, finally got this sorted today, so as you can see its been an eventful 3 day’s and I’m shagged to be honest.

Now if the power supply arrives for the BDM Server arrives tomorrow the repair jobs will be over and done with and me and Tony can finally relax.

Its days like these that hopefully make people realise there’s a reason for having an IT Dept, ok we may have days where we do fcuk all, but when were needed we deliver 🙂

Anyway hopefully touch wood there won’t be anymore disasters for a while and I can go back to sitting on my hands