(I don’t envy server engineers. As a mostly-client engineer, I can take down the local device, and that’s very bad, especially with app auto-updates. But I can’t usually take down a dozen other things as collateral damage! #HugOps)

(I don’t envy server engineers. As a mostly-client engineer, I can take down the local device, and that’s very bad, especially with app auto-updates. But I can’t usually take down a dozen other things as collateral damage! #HugOps)

Real-time photo of #Cloudflare #DevOps trying to get their service back up and running while web requests keep pouring in.
#HugOps

#hugops to @github

Real-time photo of #Cloudflare #DevOps trying to get their service back up and running while web requests keep pouring in.
#HugOps

@jimsalter @joeress here's one for your 2.5 admins. #25admins #zfs #postgresql

https://matrix.org/blog/2025/10/post-mortem/ great write-up and #HugOps to those involved over that very stressful period.

- it's great to see they had a comprehensive backup/recovery strategy in place
- well done, really well done. Multiple fallback layers. Sad that they had to use them, but hey that's why we do this.
- Kudos to @beasts hosting for moral & technical support, once again I keep hearing good things about them

A story in SQL backup/recovery from matrix.org with three key lessons:

- always do critical recovery work with 2+ people checking and reviewing together (they did this), and rotate regions because sleep is critical
- never actually delete stuff during a crisis. Ever (narrator: they learned this the hard way)
- ZFS would have made this recovery significantly easier, in so many ways

It would have been almost trivial to recover from their failed storage with ZFS, and perhaps avoid either the failover, or the remote restore.

Scheduled ZFS snapshots would have meant a rollback instead of a recover in at least 2 of their high-risk moments.

It would have also meant higher storage costs because snapshots are almost but not quite free.

ZFS snapshots can be sent/received from an alternate system, over LAN at very high rates, much much faster than a remote S3-based streaming restore.

#hugOps story time! Quote this and tell me the biggest incident you ever saw in production. It’s inevitable, it’s gonna happen, and learning from incidents is way better than shitting on people trying to fix them.

I’ll start :)

https://hazelweakly.me/blog/mother-of-all-outages/

#hugOps story time! Quote this and tell me the biggest incident you ever saw in production. It’s inevitable, it’s gonna happen, and learning from incidents is way better than shitting on people trying to fix them.

I’ll start :)

https://hazelweakly.me/blog/mother-of-all-outages/

Did AWS join the government shut down?

#USpol #AWS #GeekHumor #HugOps

I am seeing *so many* bad takes about the AWS outage, so many.
Everybody is smug until they get fucked with their pants on.
Even if you've done everything to run your own stuff and host it, it is very difficult to avoid services that will be impacted and there's *no way* the majority of businesses are running all that stuff themselves.
Comms tools, status pages, payment systems, monitoring systems, build tool, deployment tools, planning tools, the list goes on.
If you're telling me you're running absolutely everything yourself, well done, I have no idea how, and if you're better at all of that than all the SAAS providers, I struggle to believe you've time left in the day to run your actual business.
100% you should be doing your due diligence to make sure you're resilient where it matters, but being caught up in something like this is almost inevitable if you're a non trivial online company.
Stop throwing rocks, start sending #hugOps

https://health.aws.amazon.com/health/status has a fascinating stream of updates if you are into that kind of thing. #hugops for everyone involved, it looks like an uphill battle. #aws

https://health.aws.amazon.com/health/status has a fascinating stream of updates if you are into that kind of thing. #hugops for everyone involved, it looks like an uphill battle. #aws

#HugOps to all the IT and business people whose start of week has been borked by the AWA outage, while they try to restart everything from the ground up.

#HugOps to all the IT and business people whose start of week has been borked by the AWA outage, while they try to restart everything from the ground up.

Massive #HugOps to AWS and their ecosystem. Outages are stressful, let alone international ones, so let's be extra kind to each other today

I see everyone is getting their unscheduled chaos monkey tests. #hugops

BIIIIG #hugops to ... the whole internet? tonight, but especially to @signalapp

#hugops to literally the whole Internet 🥲

#HugOps to everyone using Node.

https://www.stepsecurity.io/blog/ctrl-tinycolor-and-40-npm-packages-compromised

FINASWAN

@Floppy Is Nice And So We Are Nice 💜

#Ruby #RubyOnRails #HugOps