10

While diagnosing low memory problems in my desktop machine (details at U&L) I've noticed that my noncache "kernel dynamic memory" is big:

# smem -twk
Area                           Used      Cache   Noncache 
firmware/hardware                 0          0          0 
kernel image                      0          0          0 
kernel dynamic memory          1.1G     369.3M     801.7M 
userspace memory               2.0G     133.3M       1.9G 
free memory                  734.1M     734.1M          0 
----------------------------------------------------------
                               3.9G       1.2G       2.7G

On two other systems I checked it was 150MiB (also a desktop, but with 8GiB or RAM) and 29MiB. Nowhere near the 20% of my desktop machine.

How can I find out what makes it this large?

BTW: I already checked smem sources, it basically does (memtotal - userspace - free - cache).

/proc/meminfo:

# cat /proc/meminfo 
MemTotal:        4051956 kB
MemFree:          508276 kB
Buffers:           35232 kB
Cached:           651052 kB
SwapCached:       121380 kB
Active:          1358008 kB
Inactive:        1351596 kB
Active(anon):    1184616 kB
Inactive(anon):   886904 kB
Active(file):     173392 kB
Inactive(file):   464692 kB
Unevictable:        8616 kB
Mlocked:            8616 kB
SwapTotal:       4051952 kB
SwapFree:        3815780 kB
Dirty:               348 kB
Writeback:             0 kB
AnonPages:       1971164 kB
Mapped:           140108 kB
Shmem:             44656 kB
Slab:             176564 kB
SReclaimable:      62080 kB
SUnreclaim:       114484 kB
KernelStack:        3352 kB
PageTables:        43012 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     6077928 kB
Committed_AS:    3681164 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      139780 kB
VmallocChunk:   34359570976 kB
HardwareCorrupted:     0 kB
AnonHugePages:    448512 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:     2536128 kB
DirectMap2M:     1656832 kB
1
  • You should not trust smem perhaps, see this question: unix.stackexchange.com/questions/59450/swap-usage-too-high If you look at the output of meminfo it says that Shmem (shared memory) is about 700 MB. If you feed that meminfo to smem, it reports Kernel cache memory has beeing slightly more than 700 MB, smem counted the Shmem within kernel memory, whereas this is IMHO user space memory!
    – Huygens
    Mar 4, 2013 at 7:57

2 Answers 2

4

Seeing another of your post I guess you are using zram. So that will be my assumption here.

I did the experience to install zram and consume lot of memory, and I got the same output of smem than you. smem does not take into account zram into its counting, it only uses /proc/meminfo to compute its value, and if you look and try to understand the code you will see that the zram RAM occupation is gets in the end counted under the noncache column of the kernel dynamic memory line.

Further investigations

Following my gut feeling that zram was behind this behavious, I setted up a VM with similar spec as your machine: 4 GB RAM and 2 GB zram swap, no swap file.

I have loaded the VM with heavy weight applications and got the following state:

huygens@ubuntu:~$ smem -wt -K ~/vmlinuz-3.2.0-38-generic.unpacked -R 4096M
Area                           Used      Cache   Noncache 
firmware/hardware            130717          0     130717 
kernel image                  13951          0      13951 
kernel dynamic memory       1063520     922172     141348 
userspace memory            2534684     257136    2277548 
free memory                  451432     451432          0 
----------------------------------------------------------
                            4194304    1630740    2563564 
huygens@ubuntu:~$ free -m
             total       used       free     shared    buffers     cached
Mem:          3954       3528        426          0         79        858
-/+ buffers/cache:       2589       1365
Swap:         1977          0       1977

As you can see free reports 858 MB cache memory and that is also what smem seems to report within the cached kernel dynamic memory.

Then I further stressed the system using Chromium Browser. At the beginning, it was only have 83 MB of swap used. But then after a few more tabs opened, the swap switch quickly to almost it's maximum and I experienced OOM! zram has really a dangerous side where wrongly configured (too big sizes) it can quickly hit you back like a trebuchet-like mechanism.

At that time I had the following outputs:

huygens@ubuntu:~$ smem -wt -K ~/vmlinuz-3.2.0-38-generic.unpacked -R 4096M
Area                           Used      Cache   Noncache 
firmware/hardware            130717          0     130717 
kernel image                  13951          0      13951 
kernel dynamic memory       1355344     124072    1231272 
userspace memory             961004      36456     924548 
free memory                 1733288    1733288          0 
----------------------------------------------------------
                            4194304    1893816    2300488 
huygens@ubuntu:~$ free -m
             total       used       free     shared    buffers     cached
Mem:          3954       2256       1698          0          4        132
-/+ buffers/cache:       2118       1835
Swap:         1977       1750        227

See how the kernel dynamic memory (columns cache and non-cache) look like inverted? It is because in the first case, the kernel had "cached" memory such as reported by free but then it had swap memory held by zram which smem does not know how to compute (check smem source code, zram occupation is not reported in /proc/meminfo, this it is not computed by smem which does simple "total kernel mem" - "type of memory reported by meminfo that I know are cache", what it does not know is that in the computed total kernel mem it has added the size of the swap which is in RAM!)

When I was in this state, I activated a hard disk swap and turned off the zram swap and I reset the zram devices: echo 1 > /sys/block/zram0/reset.

After that the noncache kernel memory melted like snow in summer and returned to "normal" value.

Conclusion

smem does not know about zram (yet) maybe because it is still staging and thus not part of /proc/meminfo which reports global parameters (like (in)active pages size, total memory) and then only report on a few specific parameters. smem identified a few of this specific parameters as "cache", sum them up and compare that to total memory. Because of that zram used memory gets counted in the noncache column.

Note: by the way, in modern kernel, meminfo reports also the shared memory consumed. smem does not take that yet into account, so even without zram the output of smem is to consider carefully esp. if you use application that make big use of shared memory.

References used:

7
  • deactivating zram doesn't help (yes, it does reduce the amount of kernel memory, but only by about 100MiB), the 5 biggest processes took about 300 (X) to 50MiB RSS, in total, less than 600MiB. Mar 2, 2013 at 19:51
  • Could you use slabtop if available and see if any kernel activity could be responsible for the memory usage? I will try to investihate further later today what is exactly calculting smem and if it makes sense at all!
    – Huygens
    Mar 3, 2013 at 8:38
  • I have setup a VM with 4 GB RAM and only 2 GB swap using zram. I opened a few heavy-weight applications: Firefox (many tabs with pictures, facebook pages, etc.), LibreOffice, Gimp with several 10MP photos. The output of smem was 1 GB kernel (more or less everything cache) and 2.5 GB userspace (more or less everything non-cache). There was no swap consumed. Then I launched Chrome, opened a few pages, I started to have some swap 83 MB, but that did not change things much for smem. However, I opened several other tabs and bam! I had a similar output to smem, see my updated answer.
    – Huygens
    Mar 3, 2013 at 18:02
  • slabtop reports about 200-300MiB memory in total. While with running the applications I can make smem report similar numbers on my other machine by running applications, turning the applications off does decrease the kernel dynamic memory on the other machine. On the problematic machine, the noncache kernel dynamic memory just keeps on growing until the system is unusable (about a week of uptime) Mar 3, 2013 at 21:11
  • @HubertKario Maybe zram is partially only the culprit, but I have this gut feeling that when your system piles up too much of "this memory" you are looking after, then zram might just makes matter worse :) But don't get me wrong, I am trying to help here. Have you try running your server for a week without zram (but 4 GB hard disk swap)? Do you hit the same problem? Sooner? Later? As I told you, smem could wrongly compute the noncache kernel memory, it might not be all kernel it might not be all noncache. Trust ps, /proc/<pid>/ and /proc/meminfo.
    – Huygens
    Mar 3, 2013 at 21:38
0

Doesn't look bad to me, plenty of easily reclaimable memory around. Exactly what isn't working (not just "Oh, horror, look at the numbers <random program> gives!")? Programs crash (OOM, out of memory, handler kicking in)? Programs don't start? System feels sluggish? Incessant disk activity? Any clues in the logs?

Linux will fill up all available memory, it is cheaper just to keep stuff around than to actively erase it, and it might be used again later. A quiescent machine (or just after boot) will give very different numbers than an actively used one.

1
  • 2
    The system becomes sluggish. When I'm running KMail (so Akonadi, Virtuoso), Opera, Amarok, Ktorrent and few other applications I'm left with less than 700MiB for file system cache when kernel hogs to itself a full 1GiB, hardly optimal. And this is only after X has been running for a week or so. Simple logout and login brings the system back to optimal performance (with exactly the same applications running). Logs led me to a bug related to readon driver, I'm currently testing if the issue is still present. Jan 29, 2013 at 16:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.