New blog post: "The RAM Nightmare: How I Lost My Sanity (and Almost My Deadline)": https://www.davidrevoy.com/article1117/the-ram-nightmare-how-i-lost-my-sanity-and-almost-my-deadline
Discussion
Loading...
@davidrevoy It seems to me that operating systems could and should detect bad memory, but sadly a lot of software is built without fully taking into account the fact that hardware fails.
Relevant fact: Linus Torvalds is an advocate of error-correcting memory (https://arstechnica.com/gadgets/2021/01/linus-torvalds-blames-intel-for-lack-of-ecc-ram-in-consumer-pcs/) and uses it on his own machine (https://www.youtube.com/watch?v=mfv0V1SxbNA).
@Changaco @davidrevoy Most of the software-based means of detecting bad memory have the failure case of being unable to do much about faults that develop while the machine is already on & booted, with handling having to wait until next reboot.
Doing more without considerable drawbacks requires hardware support.
Presumably one could make a VM runtime that does all sorts of parity calculation shenanigans but the performance impact would be prohibitive.
Doing more without considerable drawbacks requires hardware support.
Presumably one could make a VM runtime that does all sorts of parity calculation shenanigans but the performance impact would be prohibitive.
@lispi314 I'm not sure what you mean. It seems to me that the kernel could regularly check memory pages in the background, stop using memory address ranges that return corrupted data, and of course emit a warning meant to be relayed to the user by its desktop environment. Personal devices rarely max out their hardware capabilities, so there are plenty of times when background checks like this can be run without significantly impacting performance.
@Changaco @davidrevoy It would have to check the address range *every* time before any allocation at minimum to be somewhat reliable, and that actually wouldn't cover the memory where the code to do those checks in the kernel is itself stored.
Alternatively triple allocation, lookup and so on with on-the-fly comparison.
But that still runs into the issue of corruption of the immediate check storage.
If CPUs exposed the cache explicitly and one trusted them to never get corrupted or flip (why such trust?) then one could have the core memory-checking runtime stored there.
Funny thing there is ECC memory somewhat addresses the case of memory corruption but the case of in-CPU corruption? That takes mainframe-grade CPUs to handle correctly, everything else likes to pretend CPUs are perfect and never do anything wrong (which has been empirically demonstrated to be very much false).
Alternatively triple allocation, lookup and so on with on-the-fly comparison.
But that still runs into the issue of corruption of the immediate check storage.
If CPUs exposed the cache explicitly and one trusted them to never get corrupted or flip (why such trust?) then one could have the core memory-checking runtime stored there.
Funny thing there is ECC memory somewhat addresses the case of memory corruption but the case of in-CPU corruption? That takes mainframe-grade CPUs to handle correctly, everything else likes to pretend CPUs are perfect and never do anything wrong (which has been empirically demonstrated to be very much false).
@lispi314 I didn't mean the kernel should try to completely prevent memory corruption by doing software-based ECC on all pages all the time. I meant the kernel should at least try to detect faulty memory as soon as possible, without significantly impacting performance, instead of doing nothing to address a rare but real problem that affects end users.
@Changaco @davidrevoy I suppose it could, though it would require the kernel to cope with periodic moving/rezoning of its memory to truly do properly.
I think device drivers (and firmware) might complain about it, depending on how they're mapped with kernel memory.
For userspace programs virtual memory mapping makes this a lot easier though.
1 more replies (not shown)
@Changaco Thank you for the links! Interesting read, and I totally agree. I had no idea ECC was a thing before this article, but now, I even don't understand why this tech is not the standard. RAM problems degenerates in so many troubles...
@davidrevoy It occurs to me that if you bought your RAM in 2020 it's unlikely to be affected by demand for AI kit spiking the price of DDR5—it'll be an older type of module. So should still be available second-hand for not too much more money.
@cstross @davidrevoy It's DDR4, where the prices have also gone up unfortunately.
I saw a DDR4 kit I bought for $60 spike to $250. Not sure if it was increased demand because DDR5 got harder to find or some seller trying to take advantage of the confusion though
As far as I know, DDR4 RAM has also spiked as people try to upgrade older motherboards rather than buy new ones. People have been buying up old hardware, stripping the RAM out of it, putting it together into a smaller number of mobos and selling those in the used market, leaving a lot of older DDR4 mobos as waste.
I don't have a link right now, but have read a bit about these things via mastodon links in the last few weeks.
@cstross 🤩 Oh nice! I'll check it. I still haven't done a single web search on the topic, convinced it would be too expensive anyway for a quick replacement 'on the fly'.