jbz
jbz boosted

Once again today, #FreeBSD and #ZFS saved a setup. Suddenly, a colleague realized that a database was acting up - probably some massive operation had deleted something. The machine takes snapshots every 15 minutes and keeps them for a few hours, then one a day and keeps those for days. To make a long story short, the July 4th dump still had the correct data. To get there, we just had to clone all the snapshots (going back day by day) and test them.

Snapshots are one of the best inventions since sliced bread.

#RunBSD#OpenZFS

#ZFS question: if I've taken a recursive snapshot, is there a straightforward way to make a recursive clone of that snapshot?

# zfs snap -r pool/path/to/dataset@test
# zfs list -o name -Hr -t snap pool/path/to/dataset
pool/path/to/dataset@test
pool/path/to/dataset/subdataset1@test
pool/path/to/dataset/subdataset2@test

Great. so now I want to clone that:

# zfs clone pool/path/to/dataset@test pool/path/to/other

# zfs list -o name -Hr pool/path/to/other
pool/path/to/other

There's no pool/path/to/other/subdataset1 or pool/path/to/other/subdataset2

In other cases, the ZFS commands take `-r` to do things recursively, but it looks like `zfs clone` doesn't accept the -r option.

So I can manually clone all those sub-dataset snapshots into the new clone, but is there a way to have the zfs-clone do it all in one go? (on FreeBSD 14.2 if that matters)

Dev just messaged me, alarmed: a WordPress plugin's tech support logged into his site and broke everything! 😱
But Dev is sharp, and even at 22:10, he remembered his server runs FreeBSD and ZFS. With snapshots of his site and database every 15 minutes, we rolled back to the 20:00 snapshot, and his site was back up in a flash!

Thank you, FreeBSD! Thank you, ZFS!

#FreeBSD#ZFS#WebDev#WordPress#SysAdmin#DataRecovery#Snapshots#ThankYouTuesday

New blog post!

I describe how I'm using the snapshot feature of ZFS to track daily changes in my home directory on FreeBSD. This enables me to do things like restore files accidentally deleted and make backups to remote devices.

I have quickly come to appreciate how useful snapshots can be:

https://www.dwarmstrong.org/zfs-snapshots/

#ZFS#FreeBSD#RunBSD

You can add a remote disk to a #ZFS mirror using ggatec(8) on #freebsd and the the ZFS pool happily resilvers to the remote disk connected via #geomgate.

I added the GEOM Gate device to the #ZFS mirror because I wanted to switch disks. On the screenshot, ada1 only had 128GiB and the disk behind ggate0 had 1TiB and was the one I wanted to use in the future. I prepared ada1 with “zpool set autoexpand=on mypool” and “zpool online -e mypool ada1”. Then I added the 1 TiB disk to create the mirror. When ggate0 was done with the resilvering, I detached ada1, powered the two machines involved off and physically switched disks. I’m wondering if that works just as good when the machine in question is still running.

Today, out of nowhere, my shell started to misbehave. My prompt suddenly reverted to default. Some unexpected "command not found" errors started popping up. Something was off.

I went to check my shell configuration. The directory was not there. I then went look into ~/.config. Half of the directories inside were simply gone.

I immediately flipped into what the fuck is happening mode.

This system is an Alpine root-on-ZFS. I have a script called by cron every 20 minutes that snapshots everything.

First I went into the snapshot directory and started copying some things. I soon noticed it was just too much missing. How to map out what was gone in the first place? Even so, copying would only go so far because I was duplicating things.

I looked to my left at the resource monitor. I had less than 1 GB left of free space. That was not going to work.

I flip some pages, looking for an incantation...

% zfs rollback zroot/home@20m

A long second hanged in the air. Then all the resource monitor's bars flipped at once to green: 53% free space.

"Blessed be the ZFS Daemons and the Authors who conjured Them."

The system was still confused, so I rebooted. It let its conscience drift - as it is used to -, uncertainty still heavy in the air. Then it resurfaced... every line of output unconcerned.

Back up, no trace was left of the seconds leading up to the warp. The only suspicion came from a cryptography guardian, who noticed something was wrong about the timestamps. "Please re-enter the passcodes", it asked. Every other blob was either unconcerned or unimpressed with the glitch.

Like any time travel, the only trace left was in my memory. No history anywhere has me looking for that spell. I booted 20 minutes into the past and that's from when I am writing to you.

#ZFS