The Matrix.org Foundation
@matrix@mastodon.matrix.org  ·  activity timestamp 4 days ago

Sorry, but it's bad news: we haven't been able to restore the DB primary filesystem to a state we're confident in running as a primary (especially given our experiences with slow-burning postgres db corruption). So we're having to do a full 55TB DB snapshot restore from last night, which will take >10h to recover the data, and then >4h to actually restore, and then >3h to catch up on missing traffic. Huge apologies for the outage. Again, folks using their own homeservers are not impacted.
Marcos Dione
@mdione@en.osm.town replied  ·  activity timestamp 3 days ago
@matrix just an idea to improve backups:

Make exponential backoff like backups: last month, months 2-3 ago, mos 4-6 ago, 7-12moa, 2-3ya, etc. Or with N messages instead of N days.

Sounds like you could recover the fresher data first, then catch up, then restore backwards.

#backup#SysAdmin

scribe
scribe boosted
Julian Oliver
@JulianOliver@mastodon.social  ·  activity timestamp 6 days ago

YunoHost (https://yunohost.org/) comes into my feed with a wrapping of excitement from time to time.

It's a nice idea that people should not need skill & understanding to deploy a complex service, but know this: deployment is just the beginning (and often the easy bit). Complex services and the servers they run on need to be sufficiently understood before they can be secured, maintained debugged, patched and even migrated. This requires knowing what goes where, and why.

1/n

#sysadmin
Julian Oliver
@JulianOliver@mastodon.social  ·  activity timestamp 6 days ago

YunoHost (https://yunohost.org/) comes into my feed with a wrapping of excitement from time to time.

It's a nice idea that people should not need skill & understanding to deploy a complex service, but know this: deployment is just the beginning (and often the easy bit). Complex services and the servers they run on need to be sufficiently understood before they can be secured, maintained debugged, patched and even migrated. This requires knowing what goes where, and why.

1/n

#sysadmin
h3artbl33d :openbsd: :ve:
@h3artbl33d@exquisite.social  ·  activity timestamp last week
Sigh. So apparently Microsoft bounces e-mail to a non-existing recipient with 50 5.4.1 Recipient address rejected: Access denied.flan_molotov

How vague could they be? Something like "550 Requested action not taken: mailbox unavailable" or "550 5.1.1 Mailbox "nosuchuser" does not exist" are clear and standardized.

#email #mailops #sysadmin
h3artbl33d :openbsd: :ve:
@h3artbl33d@exquisite.social  ·  activity timestamp last week
Sigh. So apparently Microsoft bounces e-mail to a non-existing recipient with 50 5.4.1 Recipient address rejected: Access denied.flan_molotov

How vague could they be? Something like "550 Requested action not taken: mailbox unavailable" or "550 5.1.1 Mailbox "nosuchuser" does not exist" are clear and standardized.

#email #mailops #sysadmin