@shinydan the way he tells the story, the punchline is a lesson I will not forget. “On every system we have this looks like a coordinated attack or a geopolitical event, like something massive. In reality, it’s the two most boring things you can imagine, one company cheaping out to save a few bucks and one person digging a ditch in the wrong place.”
1 Boost
0 Likes
@mhoye @shinydan a similar thing happened with the redundant power feeds to a data center I worked in:
* power utility fed them both from the same circuit somewhere upstream;
* when there was a failure of the common supply upstream, at least the diesel generator on site correctly came on;
* but the generator’s extra fuel tank with four hour capacity wasn’t plumbed in properly;
* “generator fuel low” alarm only monitored the extra tank, which never drained, so it never triggered;
* on site batteries were only specced for a 15 minute run time because of the assumptions that the power supply was really redundant and the generator would always be refueled before it ran out.
Any single one of those problems on its own should have been survivable, but a high capacity power line failed due to heat sag during what happened to be a long holiday weekend, the generator only ran for like 45 minutes, the batteries lasted not ten minutes after that, and the entire data center then went dark for several hours.
* power utility fed them both from the same circuit somewhere upstream;
* when there was a failure of the common supply upstream, at least the diesel generator on site correctly came on;
* but the generator’s extra fuel tank with four hour capacity wasn’t plumbed in properly;
* “generator fuel low” alarm only monitored the extra tank, which never drained, so it never triggered;
* on site batteries were only specced for a 15 minute run time because of the assumptions that the power supply was really redundant and the generator would always be refueled before it ran out.
Any single one of those problems on its own should have been survivable, but a high capacity power line failed due to heat sag during what happened to be a long holiday weekend, the generator only ran for like 45 minutes, the batteries lasted not ten minutes after that, and the entire data center then went dark for several hours.
@mhoye @shinydan
years back, i worked at an ISP using SPRINT for long haul. we'd paid a serious premium to have "geographically diverse" cross US fiber.
then we had an outage that took out both routes at once.
turns out, they'd regroomed without telling us and "diverse" meant on both sides of the same train track in the same train yard. then a derailment took out both at once.