Consider the following scenario. You have a slow read-only media (e.g. write-protected Thumb Drive, CD/DVD, whatever) that you installed Linux on (not a Live CD per se, but a normal build), and use it on a computer with literally no other forms of storage. It's slow, because it is USB 2. The root filesystem is mounted as overlayfs so that it's "writeable" for logs and a lot of other temporary work you do, but all writes go to RAM (tmpfs upperdir). Pretty typical scenario for a Live distro situation.
Since there is no other forms of storage, swap is mounted on zram. So when Linux decides to swap, it compresses those pages and stores them still in RAM, but at least they're compressed. This is actually decent, since the RAM of most applications is easily compressible (RAM is usually very redundant in data since it's meant to be "fast"). This works well for application memory, but not for tmpfs.
Here's the thing: zram is fast, incredibly so. The Thumb Drive, on the other hand, is slow. Let's say it's 20 MiB/s, which is really slow in comparison. You can see the problem and why the kernel will not do the right thing here.
Note that this question is not a duplication of How to make files inside TMPFS more likely to swap. The question is pretty much the same, but I'm not satisfied with that answer in that question whatsoever, sorry. The kernel definitely does not do the "right thing" by itself, regardless of how smart the people designing it are. I dislike it when people don't understand the situation and think they know better. They cater to the average case. That's why Linux is so tweakable, because no matter how smart it is, it can't predict what it will be used for.
For example, I can (and did) set vm.swappiness (/proc/sys/vm/swappiness) to 100, which tells it to swap application memory aggressively and keep the file cache. This option is nice, but it's not everything, unfortunately.
I want it to prioritize keeping the file cache over any other RAM use when dealing with swap. That's because dropping the file cache results in it having to read back from the slow 20 MiB/s drive, which is much much slower than swapping to zram. For applications, the vm.swappiness works, but not for tmpfs.
tmpfs is mounted as page cache, so it has the same priority as the file cache. If you read a file from tmpfs, it will prioritize it over an older file cache entry (most recently used). But that's bad, the kernel clearly does not do the right thing here. It should consider that swapping tmpfs to zram is much better even if it's "used more recently" than the file cache because reading from the drive is very slow.
So I need to explicitly tell it to swap from tmpfs more often compared to the file cache: that it should preserve the file cache more than tmpfs. There are so many options in /proc/sys/vm but nothing for this that I could find. Disappointing really.
Failing that, is there a way to tell the kernel that some devices/drives are just that much more slower than others, and it should prefer to keep the cache for them more than others? tmpfs and zram are fast. The thumb drive is not. Can I tell the kernel this information?
It can't do "the right thing" by itself if it treats all drives the same. It's much faster to swap tmpfs to a fast drive like zram than to drop caches from a slow drive, even if tmpfs is used more recently.
When it runs out of free memory it will start to either swap application memory (good) due to swappiness, or drop old file caches (bad). If I end up re-reading from those files, it will be very slow. Much slower than if it decided to swap some tmpfs, even if recently used, and then read from it again. Because zram is an order of magnitude faster.