1

I have my home server with Proxmox installed. During installation I created just a default swap partition of small size, but did not consider it necessary, since my server has 192GB of RAM installed. However, I was surprised, swap is getting used from time to time. It is not rare situation, when system has 60+GB of memory still available and just a few hundreeds of megs of swap used. Even more, for experiment I decided to disable swap. As a result, I got some of my VMs sporadically killed, while monitoring tools report enough of resources in system. Since I am running pretty simple setup without much tweaking, I realized swap is essential for hypervisors somehow.

Now the good question is how to create proper swap setup.

I have 4 NVME SSDs in my system. I plan to create a ZFS pool with them and use resulting pool for fast storage needs. There are some ideas how to create swap with ZFS, but it seems a bad decision. ZFS allocates memory during writes and this could lead to failures.

Another idea is to split NVMEs into partitions, one small one is for swap, another big one is for ZFS data. I did experiment. I created 16GB swap partition on every drive and with the rest of space from each drive created ZFS pool. After this I configured kernel to use 4 swap partitions. Everything works as expected, however I started to observe, that kernel is using only one of the SSDs for swapping, leaving all other without much utilization.

So, how to balanse load across multiple SSDs in such case?

3
  • 1
    "So, how to balanse load across multiple SSDs in such case?" sounds to me like the balancing wouldn't help; the write rates are probably low enough for the single SSD to deal with it about as good as a whole array of SSDs can. Check that all swaps have the same priority, though. Sep 22, 2023 at 19:25
  • Thanks a lot for your comment. I read somewhere, that TrueNAS Scale creates RAID0 md-array from swap partitions and this way forces kernel to use different SSDs for swapping. Do you see any issues with such approach? Sep 22, 2023 at 21:02
  • 1
    that sounds like it unnecessarily adds another buffered layer, is my concern: man 2 swapon tells us the kernel will round-robin to different active swaps of the same priority. Sep 22, 2023 at 22:22

1 Answer 1

1

There is no need to raid swap disks. Doing so is not helpful and is probably harmful.

You could make a separate partition from zfs on each disk and use them all as swap. If you give them different priorities, they will be used sequentially; as each fills, the next one will start being used. You probably should give all of them the same priority, then they will be used in parallel, without need for raid.

You could instead make a swap file on zfs and zfs will handle making access parallel. However, in the past this had crash level bugs and some zfs version documentation says swap files are unsupported. Also, if your zfs is using a redundant raid, it is likely a waste.

A better zfs alternative is to create a swap volume with the zfs create -V command. This is better than separate swap partitions in that it allows you to resize the swap volume if you need to, and zfs can manage I/O completely rather than having swap and zfs compete for the disks. Also, an explicit zfs swap volume won't have issues with using ram for cache to hold swap pages to release ram....

See also https://askubuntu.com/questions/228149/zfs-partition-as-swap

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .