6

Consider the following scenario. You have a slow read-only media (e.g. write-protected Thumb Drive, CD/DVD, whatever) that you installed Linux on (not a Live CD per se, but a normal build), and use it on a computer with literally no other forms of storage. It's slow, because it is USB 2. The root filesystem is mounted as overlayfs so that it's "writeable" for logs and a lot of other temporary work you do, but all writes go to RAM (tmpfs upperdir). Pretty typical scenario for a Live distro situation.

Since there is no other forms of storage, swap is mounted on zram. So when Linux decides to swap, it compresses those pages and stores them still in RAM, but at least they're compressed. This is actually decent, since the RAM of most applications is easily compressible (RAM is usually very redundant in data since it's meant to be "fast"). This works well for application memory, but not for tmpfs.

Here's the thing: zram is fast, incredibly so. The Thumb Drive, on the other hand, is slow. Let's say it's 20 MiB/s, which is really slow in comparison. You can see the problem and why the kernel will not do the right thing here.

Note that this question is not a duplication of How to make files inside TMPFS more likely to swap. The question is pretty much the same, but I'm not satisfied with that answer in that question whatsoever, sorry. The kernel definitely does not do the "right thing" by itself, regardless of how smart the people designing it are. I dislike it when people don't understand the situation and think they know better. They cater to the average case. That's why Linux is so tweakable, because no matter how smart it is, it can't predict what it will be used for.

For example, I can (and did) set vm.swappiness (/proc/sys/vm/swappiness) to 100, which tells it to swap application memory aggressively and keep the file cache. This option is nice, but it's not everything, unfortunately.

I want it to prioritize keeping the file cache over any other RAM use when dealing with swap. That's because dropping the file cache results in it having to read back from the slow 20 MiB/s drive, which is much much slower than swapping to zram. For applications, the vm.swappiness works, but not for tmpfs.

tmpfs is mounted as page cache, so it has the same priority as the file cache. If you read a file from tmpfs, it will prioritize it over an older file cache entry (most recently used). But that's bad, the kernel clearly does not do the right thing here. It should consider that swapping tmpfs to zram is much better even if it's "used more recently" than the file cache because reading from the drive is very slow.

So I need to explicitly tell it to swap from tmpfs more often compared to the file cache: that it should preserve the file cache more than tmpfs. There are so many options in /proc/sys/vm but nothing for this that I could find. Disappointing really.

Failing that, is there a way to tell the kernel that some devices/drives are just that much more slower than others, and it should prefer to keep the cache for them more than others? tmpfs and zram are fast. The thumb drive is not. Can I tell the kernel this information?

It can't do "the right thing" by itself if it treats all drives the same. It's much faster to swap tmpfs to a fast drive like zram than to drop caches from a slow drive, even if tmpfs is used more recently.

When it runs out of free memory it will start to either swap application memory (good) due to swappiness, or drop old file caches (bad). If I end up re-reading from those files, it will be very slow. Much slower than if it decided to swap some tmpfs, even if recently used, and then read from it again. Because zram is an order of magnitude faster.

  • 1
    You may want to tone down that attitude. How the kernel works isn't generally the fault of unix.SE users, and the answer in the question you linked to also doesn't represent the opinions of every user reading this one. (Instead, it's just the opinion of one user, seen reasonable by six others.) – ilkkachu May 17 '18 at 15:03
  • Sorry, I only said that because far too many people are eager to mark questions as duplicates, but I did my research before and I don't like the "accepted answer" of that question. I'm not saying that the answerer is wrong, but it definitely does not answer the question, except with the last paragraph, when it's said that you "can't control it". I was hoping there was, though. – kktsuri May 17 '18 at 15:17
  • I might even agree in principle with that answer, but still value this question. This question raises a specific problem, which is interesting to reason about - even if we can't find a solution. The other question doesn't provide a very strong motivation. I think the general answer to the other question is that this is not something to worry about... and that you probably don't want to use very much of tmpfs. If you have reason to doubt either response, then that reason is the key part of your question. – sourcejedi May 17 '18 at 15:32
  • @sourcejedi: the link you posted in your answer pretty much confirms that Linux really is lacking in this regard. And I don't mean just the "force tmpfs to swap over the file cache". This is a specific problem of the more general "which devices are faster and should be prioritized for swapping" in fact. You can set priority on swaps, but that only applies between swaps. There's no way to set the "priority" (or lack of) on a file-backed cache. If a drive is slow, its file cache should have a high priority compared to a fast drive (or swap). Not 100% of course, but a high priority nonetheless. – kktsuri May 17 '18 at 15:38
  • Gah, I hope that last sentence or two of mine did not cause too much confusion or annoyance. I mean, on a typical desktop install which is using tmpfs for /tmp, you generally wouldn't want to put too much in /tmp, and you generally wouldn't have a reason to worry about swapping it. So I think the specific motivation for asking is quite important. Your question, here, specifies such a motivation. The other one did not, which is why I think it got a more general answer. – sourcejedi May 17 '18 at 16:21
3

I would be fairly confident that increasing swappiness makes the kernel less willing to swap tmpfs pages, and more willing to evict cached pages from the other filesystems which are not backed by swap. See commit 4f98a2fee8ac "vmscan: split LRU lists into anon & file sets":

Split the LRU lists in two, one set for pages that are backed by real file systems ("file") and one for pages that are backed by memory and swap ("anon"). The latter includes tmpfs.

and

https://elixir.bootlin.com/linux/v4.16/source/mm/vmscan.c#L2109

 * Determine how aggressively the anon and file LRU lists should be
 * scanned.  The relative value of each set of LRU lists is determined
 * by looking at the fraction of the pages scanned we did rotate back
 * onto the active list instead of evict.
 *
 * nr[0] = anon inactive pages to scan; nr[1] = anon active pages to scan
 * nr[2] = file inactive pages to scan; nr[3] = file active pages to scan
 */
static void get_scan_count(struct lruvec *lruvec, struct mem_cgroup *memcg,
               struct scan_control *sc, unsigned long *nr,
               unsigned long *lru_pages)
{
    int swappiness = mem_cgroup_swappiness(memcg);

...

    /* If we have no swap space, do not bother scanning anon pages. */
    if (!sc->may_swap || mem_cgroup_get_nr_swap_pages(memcg) <= 0) {
        scan_balance = SCAN_FILE;
        goto out;
    }

...

For example, I can (and did) set vm.swappiness (/proc/sys/vm/swappiness) to 100

You don't explicitly say whether you were able to quantify this as making no difference to your desired outcome. That said, given the extreme speed difference, I think you would really want a higher swappiness value:

Reconsidering swapping, LWN.net, June 7, 2016

The first step in the patch set is to widen the range of possible settings for the swappiness knob. In current kernels, it can go from zero (no swapping at all if possible) to 100 (reclaim anonymous and file-backed pages equally). Johannes raises the maximum to 200; at that value, the system will strongly favor swapping. That is a possibility nobody has ever wanted before, but fast drives have now made it useful.

...

Memory-management changes are fraught with potential hazards, though, and it is entirely possible that other workloads will be hurt by these changes. The only way to gain confidence that this won't happen is wider testing and review. This patch set is quite young; there have been some favorable reviews, but that testing has not yet happened. Thus, it may be a while before this code goes anywhere near a mainline kernel.

zram is certainly an example of a fast drive. It is explicitly mentioned in the discussion of this patch.

Unfortunately I did not find either commits of this patch, nor a v2 of this patch series.

The exact effect of swappiness is also discussed in more detail. I haven't worked out how this matches up with the older LWN.net explanation of swappiness details here.

  • That is interesting indeed, but unfortunately tmpfs is still considered "equal" to the file cache, since it's mounted as page cache, no matter the swappiness. +1 for the link though, very interesting read. – kktsuri May 17 '18 at 15:20
  • That's what I read all over, not just on stackexchange (but here as well). Also, kernel.org/doc/Documentation/filesystems/tmpfs.txt seems to somewhat confirm it. Plus, my own tests seem to confirm it. For example, in gkrellm, tmpfs storage shows up in "cached" along with all other caches rather than "used" -- the latter is the memory taken up by applications, so clearly it is different. I think tmpfs is "anonymous cache" rather than "file-backed", but it's still cache and not "application memory". And swappiness of 100 doesn't seem to effect it. I'll try more links. – kktsuri May 17 '18 at 15:26
  • I would not say that "application memory" is a well-defined term here. If you mmap() a file with MAP_PRIVATE (copy-on-write) and read from a page of that memory, you're accessing the page cache. That page will be shared with all other mappings, and accounted as "cache". However, if you write to such a page... that's when you get copy-on-write behaviour. You get a private copy of the page, which will never written back to the file. If such a private page is to be evicted from main memory, then it must be saved to swap instead. Such a page is accounted as "used". – sourcejedi May 17 '18 at 15:43
  • 1
    Edited answer to provide evidence for tmpfs pages being treated the same as anonymous pages. – sourcejedi May 17 '18 at 16:11
  • 1
    @kktsuri ah, right. This isn't in mainline. I didn't find a v2 of the patch series either. Edited. According to LWN.net, this requires a significant effort in reviewing and testing the associated code changes - which are more than just allowing the value to be increased :) - to know how they effect everything else. – sourcejedi May 17 '18 at 22:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.