6

I set up zram and made extensive tests inside my Linux machines to measure that it really helps in my scenario. However, I'm very confused that zram seems to use up memory of the whole uncompressed data size. When I type in "zramctl" I see this:

NAME       ALGORITHM DISKSIZE   DATA  COMPR  TOTAL STREAMS MOUNTPOINT
/dev/zram0                 2G 853,6M 355,1M 367,1M       4 [SWAP]

According to the help command of zramctl, DATA is the uncompressed size and TOTAL the compressed memory including metadata. Yet, when I type in swapon -s, I see this output:

Filename            Type        size     used    Priority
/dev/sda2           partition   1463292  0       4
/dev/zram0          partition   2024224  906240  5

906240 is the used memory in Kilobytes, which translates to the 853,6M DATA value of zramctl. Which leaves the impression that the compressed zram device needs more memory than it saves. Once DATA is full, it actually starts swapping to the disk drive, so it must be indeed full.

Why does zram seemingly occupy memory of the original data size? Why is it not the size of COMPR or TOTAL? It seems there is no source about that on the Internet yet, because I haven't found any information about this. Thank you!

3
  • 1
    I haven't played with zram lately, but a zero-order guess would be that what shows in /proc/swaps or swapon -s is just uncompressed sizes, because the general swapping system might not really understand compression. That would mean that DATA is the uncompressed size of the data zram is holding, while COMPR would be the compressed size. You could try to see what happens to the free memory on the system when something is swapped out to zram.
    – ilkkachu
    Jun 24, 2020 at 14:46
  • That's a good thought. I can see that zram definitely helps to get more out of available RAM, despite that these DATA values appear to contradict each other. For swapon -s it is the reported uncompressed size for sure. When I check the statistics with cat /sys/block/zram0/mm_stat, the value mem_used_max is a bit higher than the TOTAL value (but far below DATA). The zram.txt documentation says this is "the maximum amount of memory zram have consumed to store the data", so this leaves me really puzzled why the DATA value equals the actual memory usage on the zram swap devices.
    – Testerhood
    Jun 24, 2020 at 19:44
  • Actually looking at zramctl, mm_stat, swapon and /proc/swaps the numbers don't add up even close for swap on for me. /proc/swaps is 1k more than what mm_stat says for real usage. I'm beginning to think swapon is just wrong. Aug 5, 2021 at 22:34

1 Answer 1

17

So after more testing and observations, I made a few very interesting discoveries. DATA is indeed the uncompressed amount of memory that takes up the swap space. But at first glance it's very deceiving and confusing. When you setup zram and use it as swap, disksize does not stand for the total amount of memory that zram will consume for compressed data. Instead, it stands for the total amount of uncompressed data that zram will compress. So you could create a zram device with a size of 2 GB, but in practice zram will stop after the total compressed memory is around 500 - 1000 MB (depends on your scenario of course). Commands like swapon -s or Gnome's system monitor show the uncompressed data size for the zram device, just like the DATA of zramctl. Thankfully, in reality, zram does not actually use up the reported amount of memory. But this means that in practice, you actually have to create a zram disk size that equals the RAM you have + 50% to take real advantage of it and not a disk size that equals half of the RAM size, like zram-config incorrectly does. But read on to find out more.

Here is the deeper background: Why am I so sure? Because I tested this with zswap as well. I have compiled an own kernel where I lowered the file_prio value inside mm/vmscan.c compared to anon_prio (in newer Linux 5.6 kernels the variables have been renamed to fp and ap respectively). The reduced file_prio value will make the kernel not discard valuable cache memory as much anymore. By default, even with vm.swappiness at 100, the kernel discards an insane amount of cached RAM data, both in standby memory and for active programs. The performance hit with the default configuration is extreme in memory pressure situations when you actually want to make use of zram, because then you absolutely want the kernel to swap rarely used and highly compressible memory way more often. With more free memory, you have more space for cached data. Then cached data won't be thrown away at a ridiculously high rate, and Linux won't have to reread certain purged program file cache repeatedly. When testing this on classic hard drives, you can easily verify the performance impact.

Back to my zswap test: With my custom kernel, zswap got plenty of memory to compress once I hit the 50 - 70% memory mark. Gnome's System Monitor immediately shows a high swap data usage for the page partition, but oddly enough, there was no hard drive paging at all! This is of course by design of zswap, that it swaps least recently used memory on its own. But the interesting part is that the system reports such high swap usage for the swap partiton anyway, so ultimately you are limited by the size of your swap partition or swap file. Even though all memory is compressed, you have to have at least the swap size of the uncompressed data. Therefore even if in practice 4 GB of swapped memory in zswap use up only 1 - 2 GB, your swap needs to have the size of the uncompressed data size. The same goes for zram, but here the memory is at least not actually reserved. Well, unless you use zswap with a dynamically growing swap file of course.

As for zram again, there is also a very interesting detail that backs up the observation I made:

There is little point creating a zram of greater than twice the size of memory since we expect a 2:1 compression ratio. Note that zram uses about 0.1% of the size of the disk when not in use so a huge zram is wasteful.

This means that to make an effectice use of zram, you have to at least create a disk size that equals your installed RAM. Due to the high compression ratios, I would suggest to use your GB of RAM + 50%, but the quote above implies that it does not make much sense if you go above +100%. Additionally, since we have to specify a disk size that matches the uncompressed data size, it is much harder to control and predict the actual real memory usage. From the helpful official source above, we can limit the actual memory usage (which equals the TOTAL value of zramctl) with this command: echo 1G > /sys/block/zram0/mem_limit. But in reality, doing this will lock up the machine. Because the system tries to still swap to it, but zram imposes a limit, and the machine locks up with super high CPU usage. This behavior can't be intentional at all, which strengthens my impression that something about the whole story is very wonky.

To sum this up:

  • The disksize you set during zram device creation is basically a virtual disk size, this does not stand for the real RAM usage.
  • You have to predict the actual RAM usage (compression ratio) for your scenario, or make sure that you never create a zram disk size that is too large. Your current RAM size + 50% should be nearly always fine in practice.
  • The default configuration of the Linux kernel is unfortunately totally unsuited for zram compression, even when setting vm.swappiness to 100. You need to make your own custom kernel to actually make real use of this handy feature, since Linux purges way too many file caches instead of freeing up memory by swapping the most compressible data much earlier. Ironically, a helpful patch to fix this situation was never accepted.
  • Using the zram limit echo 1G > /sys/block/zram0/mem_limit will lock up your system once the compressed data reached that threshold. You are better off to limit zram usage with a well-predicted zram disksize, as it seems there is no other alternative for a limit.
3
  • 3
    I'm able to set vm.swappiness=200 now on linux-hardened 5.12.19 - this limit was introduced in linux 5.8 Aug 27, 2021 at 12:10
  • 2
    @Stuart Looking back on this two years later, that's an amazing change. So far the thing with the disksize remains true, so this is important to keep these details in mind. There is still Internet guides in 2022 who write stuff like "zram should be half of the RAM's size", which is untrue. The kernel.org zram doc is really insightful to understand this, as it doesn't indicate that at all.
    – Testerhood
    Jul 11, 2022 at 13:04
  • Very good insights. I just setup a zram swap device where the size is 1.5 times my ram. I think the reason not a lot of people are catching on to the fact that zram disk size can actually exceed RAM size, is because of the way it is described. Normally, one would expect disksize to mean the maximum amount of space the device will occupy when it actually becomes full, which is congruent to how we normally think of disk space. However, the documentation authors seemed to focus more on compression, which actually depends more on the compression algorithm chosen, than anything else.
    – smac89
    Dec 8, 2023 at 1:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .