Whats your approach for VMs (in #Proxmox or any other hypervisor) with large (1-2TB) Storage needs?
Currently I have an virtual #TrueNAS (with hardware pass-thru) which provides NFS/SMB Mounts for the other VMs. But this is not always possible depending on the hardware.
And creating large vDisks feels wrong.
@nan0 what exactly do you mean with "approach"? We have a Ceph Storage Layer but still try to keep the number of such big VMs small because they suck to handle.
@leah @nan0 Either what you do, if possible, other networked storage elsewhere, or local mounts in Proxmox 9.x. VDisks in the TB Range are terrible to use, fragile and easy too risky IMHO.
Local mounts in Proxmox are just not matured yet and not well exposed in the GUI. You have to care for backups yourselves, there’s no integration with normal backups or even PBS.
@MacLemon what i do limits the hardware to much, because of IOMMU and all that. Over network is not possible/feasible either because there is only one physical node and/or only Gigabit Ethernet available.
@leah as an example how would you store the data of an VM that needs 1+ TB storage for an opencloud/nextcloud? With the constrains listed above?
Ideale i would need a combination of Proxmox and TrueNAS without the hardware pass-thru.
@nan0 If you can boot Proxmox from NVMe you can pass through a whole SATA-Controller chip, even if onboard. Still somewhat risky though regarding hardware choice.
If you cannot put storage on a network, then I‘m afraid, Proxmox 9 VirtIOFS storage is the only option for large storage amounts. It’s local, ZFS backed, so it is fast, safe, All the ZFS benefits like compression, etc.
You just need to handle backups/replication of that yourselves.
@nan0 well, as said in the bigger scale we use ceph but this isn't working for a single node setup. The only setup that is comparable is the chaos.social setup and there we have one local VM for media storage with around 2TB and it is also in the PBS backup. So if you can live with the downsides, is there an argument against local storage?
But Maybe I still don't get the problem because the answer from @MacLemon confuses me, especially the beginning.
@leah @nan0 Sorry, Bad wording on my side. I did not mean, and world explicitly not recommend using a large vDisk on local storage provider. Not because of the local storage, but because of the vDisk, which is, IMHO, way too risky at n-TB size.
I‘d rather use VirtIOFS which is available since Proxmox 8.4.0. It is filesystem/file backed instead of blockdevice.
No risk of corrupted vDisk. Will benefit from ZFS on Proxmox. (You’re fine putting it on local-zfs.)
Makes more sense now?
