0

I have a Debian VM where after about a day (but sometimes sooner) all the swap gets used up (and consequently zabbix pesters me about it). I set swappiness to 0, but it still does this, albeit maybe a little slower.

The most interesting thing is, that if I list the swap usage of processes (either by top or a script) and add them up, I only get about 55MB instead of the 1022MB that is reported as used by free.

What could be the cause of the discrepancy between swap usage by adding up individual process swap usages and total reported swap usage?

OS: Debian 10.11
Linux: 4.19.0-18-amd64

6
  • 1
    wild guess: swapped kernelspace memory, not belonging to any process in particular? Commented Jun 27, 2022 at 14:57
  • @Marcus memory allocated by the kernel can’t be swapped out. Commented Jun 27, 2022 at 15:10
  • @StephenKitt is kernel memory automatically pinned? Commented Jun 27, 2022 at 15:19
  • 1
    @Marcus that depends on what you mean by “pinned”; I’m not sure what more to say than what’s in my previous comment. Commented Jun 27, 2022 at 15:27
  • @StephenKit Actually, is was tmpfs memory, which is managed by the kernel and not any process in particular (kdevtmpfs would be a candidate, but (as any kernel process) it does not report any memory usage)
    – P.Péter
    Commented Jul 5, 2022 at 10:22

1 Answer 1

3

top’s swap usage estimates and /proc’s VmSwap field (which is what the linked script uses) are extremely unreliable.

To get a decent estimate of actual swap usage, you need to look at /proc/…/smaps; smem can do that for you, e.g.

sudo smem -s swap -r|head

will show the top swap users, and

sudo smem -t

will show the total allocated swap in the summary line at the bottom (as accounted for by smaps).

Note that this can be inaccurate too, but it should give a better result. Swap can be accounted for multiple times, in which case it is overcounted in the summary line, and some content in swap is apparently not assigned to a single process (e.g. swapped-out tmpfs), in which case it is undercounted.

6
  • smem seems to give somewhat greater values, though it is still not enough by far. smem -t reports 247008k total swap usage when free reports 1017864k (and the previous script reports 60964k).
    – P.Péter
    Commented Jul 5, 2022 at 9:12
  • How much data do you have in your tmpfss? Commented Jul 5, 2022 at 9:19
  • You may be onto something, /run/log/journal has almost 1GB of data. It may be the thing that gets swapped out. It would be nice to have some way to know for sure, though (without actually deleting files from there).
    – P.Péter
    Commented Jul 5, 2022 at 9:23
  • I confimed /run/log/journal as the culprit by editing the journald config and rotating (and thus deleting most) journals. Still, it would be nice to have a tool to list tmpfs swap usage.
    – P.Péter
    Commented Jul 5, 2022 at 9:37
  • Yes, it would; if I come across one I’ll update this... Commented Jul 5, 2022 at 9:39

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .