99.9% Uptime?

We have been moving parts of the district over to a central VoIP phone switch located in our office at the Board of Education. It does put a importance on our servers staying up since if they are down the phones out at the buildings are effectively paper weights. That shouldn't be a problem though as we are near a 99.9% uptime (only 8 hours of down time a year) on our equipment.

That was until the perfect storm of a day hit. The district has been working to sell an old school building here that housed our office. So we had to get our fiber out of it and mounted onto a pole on the edge of the property. As part of that process they had to splice our fiber together again. This was supposed to be done and back up before school on Tuesday. When I arrived the 5/6th Center was not connected up meaning that their phones were the above mentioned paperweights. I quickly worked to get them linked back up and had them connected to the phone switch fairly quickly.

It wasn't long after that the power company lost a circuit breaker at one of its switching stations. That had the crazy effect of dropping one leg of the power coming into our office. Since the phone switch was on 240 that pretty much killed the power to it. We have been unable to get a UPS for it approved, but the State Department happened to have one here that would support it. We moved it over to that and powered it back up.

It was up for around 10 minutes when the UPS providing power to our main server switch failed, which then caused a kernel panic on the still recovering phone switch killing it again. We got our server switch back up just in time for the power to our core router to fail.

After several careful moves for plugs into the circuits that weren't off due to the power outage, we had things back up, at which time WRECC killed all the power to our building in order to reset the breakers.

Luckily, things held through this outage and when they brought everything back up we were good again.

I had previously said that the only things that would break the phones was a Kernel Crash, Cut Fiber, or some crazy power outage and on that day we proceeded to have all of them within the span of 3 hours. Lucky day for me.