Reading about a storage array used by a Kickstart project I'm following where they lost 3 drives in the span of a few seconds in an 8 drive RAID6 array. They are now waiting for a data recovery company to hopefully restore and clone the contents of one of the drives to rebuild the array. I don't know how this kind of failure is possible but it's a good reminder to make backups even when your data is in what is supposed to be a very reliable storage element.
10
Replies
·
0
Boosts
@gabrielesvelto Indeed, "RAID is not a backup".
This is why in my own setup I have important zfs datasets backed up to second onsite pool with syncoid and really important data also to a remote server.
@gabrielesvelto I regularly (every 3 months) run disk scrubbing on my Synology NAS devices as this will expose bitrot and other problems that might be silently occurring without accruing enough SMART events to cause the system to flag a drive as bad.
@gabrielesvelto I've also seem a few drive models, notably the infamous Seagate ST3000DM001 model, where some mechanical part is pretty much guaranteed to wear out after a certain number of actuations/period of use. Because all the drives tend to be purchased at the same time and writes/activity are equal across them, they're all likely to wear out at the same time.
@Infoseepage yes, that's what I thought. I know that large RAID arrays may make this worse by subjecting the disks to shared vibrations and resonant effects. So everything degrades at the same time and risks snapping at the same time.
@gabrielesvelto With that Seagate 3TB drive, the issue was the material used for the head parking ramp. Backblaze publishes failure data on all the drives they use and its a fascinating read. They wrote a special expose on those Seagate drives, because as you might imagine, they were pissed. They bought thousands of them and within three years only a few hundred of them were still operational.
@gabrielesvelto There is a similar problem with using SSD arrays in NAS boxes, where if you buy a bunch of SSDs and throw them in a NAS, because of the way SSDs work with having a certain number of write/erase cycles to the flash, the drives will all hit their endurance limit at about the same time.
@gabrielesvelto Synology has a mode for addressing this called F1, where the make it so that one drive gets significantly more writes than the others and wears out first. You then replace that drive safely and now it is the lowest write cycle drive and some other drive is the designated fall guy. The goal isn't to prevent drives from failing, it is to prevent too many from failing at the same time, so that array/data integrity is maintained.
@Infoseepage that's a really smart approach (no pun intended). In the past I was not too concerned about SSD drive but that changed recently. On my boxes I mostly used second-hand enterprise drives and those all have 90%+ of flash endurance after hundreds upon hundreds of full disk rewrite. But TLC/QLC drives on client machines are a different story. I've seen 5yo boxes with typical office user workloads (so quite light on writes) that have already lost 30-40% of the flash endurance. That's quite worse on smaller drives of course (sub 1TB) but given those are very common it made me wary when I help out friends and family.
Reminds me of a long outstanding bug / issue in Chrome which would cause excessive writes for some reason. Was causing issues with SSD's wearing prematurely 😉
