Post
I suspect that each individual company cares only whether its own services, and those on which it relies, are available.
Others going down may even be a bonus, with safety in numbers: don't blame us for a Cloudflare outage which is being reported everywhere, guv. We're victims too etc.
But, for *users*, having so many services behind the same reverse proxy, all going down at the same time, very much looks like bad decision-making.
@neil I think safety in numbers is under-appreciated, both for backside-covering and also people aren't going to assume you've been hacked if everyone else is down, whereas they will if you're the only one (and it's pretty difficult to prove that you've *not* been hacked).
@neil @pwaring
Isn't this just the new version of "no-one was ever fired for buying IBM"?
If you choose the dominant market player, and stuff goes south: "you can't blame me, it was general consensus that this was the thing to do"
If you choose to self-host, and anything at all goes even slightly wrong: "this wouldn't have happened if we had just gone with [insert relevant monopoly company]"
@neil I've seen this happen a few times, to the point I wonder if there's some kind of statistical law: that beyond a certain point, you run out of ability to reduce risk on average. After that, you can only shape whether you have a high rate of low-impact issues, or a low rate of high-impact issues.
And it does seem that way too much of the time, people just see "low rate of issues" and push for that, eventually leading to massive correlated failures
@rachelplusplus @neil always assumed that was a function of practice. Regular low stakes issues to get good at incident response means the big stuff isn't so big because everyone knows what to do.
@rachelplusplus Ooh, interesting.
@neil The average availability of most of these services is likely still much higher than they'd be able to manage on their own.
@neil It's the business model that says "let's bet the house on the internet", but then outsource and consolidate infrastructure because I can get rid of staff and don't have to buy hardware or run a data centre.
Then be surprised when some third party bit breaks and all trading stops.
@neil I said the same thing (about "safety in numbers") when CrowdStrike shattered Windows last year, so I'm glad someone else made the same observation🙂
I'm not judging anyone for not self-hosting. Self-hosting is not a panacea.
It would be a pain if both our primary and backup Internet connections went down, or there was a problem at A&A's end. If A&A ceased, I'd be rather stuck.
Everything would stop running if we have a power outage which outlasts my UPSs, before I could rush and start the generator.
If our home blew up, or burglars decided that the best thing to take was a massive locked rack full of aged cheap computers, it would cause me a problem while I reinstated things.
All of these are possibilities, and I do what I can to mitigate against them.
But at least it would just be *my* things.
I have had no reason - yet, anyway - to put anything I host behind someone else's reverse proxy.
Perhaps that day will come, although I very much hope not, because wow things would need to have gone badly wrong.
@neil you also have the benefit of having a website that serves content efficiently. Those who just throw up Wordpress with fancy plugins and themes, ever convoluted object caches, page caches and third party forward proxies are those that probably need those forward proxies to avoid it spewing hundreds of megabytes with every page (even to gibberish queries) killing the server's cpu and I/O.
> you also have the benefit of having a website that serves content efficiently
Yes, that was absolutely a design decision that I made.
@neil it has been, annoyingly, very useful at blocking some very dedicated AI scrapers hitting a couple of my sites - and although I *could* do it at the web hosting end, it's a lot more complicated (partially because I'm not running the web server myself)
The positives are rapidly dropping below the negatives, however
@neil The worst part about the cloudflare outage is that even if you *are* self-hosting you may still be fronted by cloudflare as it's the only well-known free/affordable way to either fend off the AI bot swarm or expose a behind-the-firewall service to the outside.
I'm using Cloudflare's free tunneling service and got burned by this but I don't know that there's any alternative (and setting that up myself is both even more work/expense and also more hassle and annoyance, which sucks)
@wordshaper @neil we had to drop support for customers fronting their sites with CloudFlare on our shared servers because it's not effective at stopping the AI bot swarm, and using it makes it impossible for us to do so ourselves. https://www.mythic-beasts.com/blog/2025/09/03/web-hosting-and-cloudflare/
@neil I think I agree, although I would argue that part of the point of reliability is to be up when others are not, though...
> part of the point of reliability is to be up when others are not, though
It might depend on one's perspective, I think.
The goal of reliability might be to be up to continue making money, to avoid being in breach of SLAs (if there is a penalty attached to them) and to avoid reputational damage.
And sometimes the cost/benefit analysis sides with "meh, that's reliable enough for our needs".
@neil fair. Linked: I think M&S/Co op's reputations were rescued somewhat by suffering from an attack at the same time. It made it bigger news and helped people understand why their online shopping was down/stock was unreliable etc.
A space for Bonfire maintainers and contributors to communicate
