the matrix.org homeserver is having problems: https://status.matrix.org/incidents/mm9hdm78svgv apologies for the inconvenience…
Post
Replies:
36
@matrix oh.. So, that's why it's not working. Good luck 🤞.
@matrix that page sends us to BluSky....
@matrix Thanks for the status update and all the work for getting it going again!
@matrix it's remainder that Matrix network to concentrated on matrix.org.
Use another homeservers, my dudes
Use another homeservers, my dudes
@matrix oh. that's a pity. Good luck for the repair and thanks for your work ❤️
But this is also a good reminder to use your own server. I am totally new to Matrix but started using it with my own instance (based on Synapse) from the beginning.
@matrix
We await good news
We await good news
So: the matrix.org database secondary lost its FS due to a RAID failure earlier today (11:17 UTC). Then, we lost the primary at 17:26. We're trying to restore the primary DB FS (which could be fastish), while also doing a point-in-time backup restore from last night (which takes >10h). We believe the incremental DB traffic since last night is intact however. Apologies for the downtime; folks on their own homeserver are of course not impacted.
@matrix why on earth do you link to bluesky instead of here?
@matrix as an advertisement for decentralization this is a bit harsh, but definitely effective!
(J/k, of course. Good luck with the recovery and thanks!)
@matrix What was the RAID failure? Have you considered using RAID-Z with ZFS?
@matrix never heard of a hot spare ?
Sorry, but it's bad news: we haven't been able to restore the DB primary filesystem to a state we're confident in running as a primary (especially given our experiences with slow-burning postgres db corruption). So we're having to do a full 55TB DB snapshot restore from last night, which will take >10h to recover the data, and then >4h to actually restore, and then >3h to catch up on missing traffic. Huge apologies for the outage. Again, folks using their own homeservers are not impacted.
Status update: we’re 47TB through restoring the 55TB db snapshot of the matrix.org DB, but then have to rebuild the DB and replay the subsequent 17h of DB traffic, which will take several hours. Thank you for your patience, and apologies once again for the outage.
Status update: we've restored the 55TB snapshot and subsequent incremental backups, and are about to replay the remaining traffic since the backup. There are still several unknowns, but if things go well the matrix.org instance should be back in 3-4 hours.
@matrix weirdly this feels like actually a positive example reinforcing the idea of a decentral fediverse, as other instances are unaffected. Also we had been discussing running an own instance at the @chaotikumev just before the outage.
I just wish there were such an easy, neat account migration feature like @Mastodon has. (And I guess I can't just ex- and import chats + keys and use SRV records to have a seamless migration?)
I just wish there were such an easy, neat account migration feature like @Mastodon has. (And I guess I can't just ex- and import chats + keys and use SRV records to have a seamless migration?)
@matrix
I don't think that this true, folks from other servers are not effected.
As result of the 18years policy I moved my community to other servers and now partially having issues with not shared security keys and lots of unencrytable messages.
I don't think that this true, folks from other servers are not effected.
As result of the 18years policy I moved my community to other servers and now partially having issues with not shared security keys and lots of unencrytable messages.
When you restore and old save state every key generated and shared in the meantime is gone.
Right?
@matrix Any plans to migrate away from centralized RDBMS? There are so many blob stores which can scale to petabytes and can tolerate the loss of multiple nodes without going offline.
@matrix Much love to the team. This incident is a reminder to me of how stable the service has been so far.
@matrix Godspeed, admins!
@matrix hugs for you all!
@matrix Say last part again louder for the people running their business relying on a major corporation without an IT department ;)
@matrix best wishes for the team working on the recovery!
@matrix better run it at ur own, i have very good experience with #conduit server https://conduit.rs
@matrix@mastodon.matrix.org
@matrix This screams to me as stressful 24 hours for infrastructure operators of matrix.org. Please accept complimentary hugs 
@matrix jokes aside, RAID failures are NOT fun. Props for the quick reaction and godpseed!
![[matrix] oopsie woopsie](https://static.piaille.fr/media_attachments/files/115/136/399/197/886/781/original/47977f81a6d6ac81.png)
@matrix This is why we need more decentralization which happens to be the goal of matrix.
@matrix good luck on the remediation actions 🫡
@matrix extra awkward when you want people to pay for it lol
