You looked at a list of processes and their memory usage. But there was a problem. You were not looking at the complete list.
gnome-system-monitor only shows "My Processes" by default. To see processes owned by all system users, including the
root user, click the menu icon in the top right (three dots in a vertical line). Change the selection from "My Processes" to "All Processes".
- Used v.s. available system RAM, as a single number
- Analyzing /proc/meminfo
- Other tools you could use - install
atop today 8-)
1. Used v.s. available system RAM, as a single number
I see your ruby code subtracts
MemTotal. This is the exact same calculation used by
gnome-system-monitor, when it shows the system is using "1.5 GiB (41.4%) of 3.7 GiB".
You are right to use either
gnome-system-monitor or your manual calculation, at least as a first approximation. The
MemAvailable number includes basically all the reclaimable "cache". I.e.
MemAvailable includes a type of "cache" that is "available" to be reclaimed, as soon as a program asks for more memory than you have free.
Side note: There is another type or meaning of "cache", which is not reclaimable. When you look at the
Cache / "cache" number, it is usually reported as including
Shmem / "shared". The
Shmem part is not reclaimable cache. The confusion happens because
Shmem was cleverly implemented using the kernel "page cache".
Another way to quickly check "available" is
free command also shows "shared", swap usage etc. Your system documentation should list the output fields and available options, i.e. in
man free. Some of the other fields can be misleading:
- The "used" field in the
free command (currently) does not include "shared". This can be very confusing. Ignore the "used" field.
- The "cached" value shown by
free suffers from the problem noted above.
free does not show "available", your system is ancient. Consult your ancient documentation.
2. Analyzing /proc/meminfo
Thank you for the full output of
cat /proc/meminfo. This often helps to find specific answers (or any answer at all). If you wanted to look through how
MemAvailable is calculated yourself, you could read the first section of my answer here: Is "Cached" memory de-facto free?
In your example
meminfo, you have
AnonPages: 924844 kB (0.9 GB).
AnonPages is one of the terms that reduces
AnonPages is increasing, that should indicate an increase in the "RES", or "RSS" ("Resident" in ram "Set Size") of some number of running programs. But RSS can be misleading, because some memory is shared:
You cannot add up RSS, because that would double-count shared memory. You have to add up PSS, the proportional RSS after accounting for sharing. The
smem command can show PSS, and also calculate totals. For example:
sudo smem -t > p; head -n1 p; echo; tail -n17 p - Look at per-process memory usage. The
tail part will show the top 15 processes, followed by a line with total PSS etc.
smem -t -U ^sourcejedi$ > U; head -n1 U; echo; tail -n17 U - Look at memory usage of processes belonging to my user, "sourcejedi".
sudo smem -t -u - Look at memory grouped by user. This might be useful to distinguish your login session(s), from some system daemons that run as their own user. E.g. packagekitd runs as the
root user, and can use hundreds of megabytes.
smem -t -P firefox - Look at the memory usage of my web browser :-).
sudo smem -t -m > m; head -n5 m; echo; tail m - Look at memory grouped by the mapping name - the name of the cached file, or "<anonymous>" or "[heap]".
Process "resident" memory includes both "anonymous" memory, and some cached files.
smem -m cannot show all cached files, only certain types of files that are currently in use. Specifically, files which a program has mapped into virtual memory. This includes program code, library code, and files mapped using mmap().
You also have
Shmem: 374848 kB (0.4 GB). I mentioned
Shmem / "shared" above. This is another term that reduces the "available" memory. (It is not reclaimable cache). This is fairly normal, but you could try to see what it is.
Some shared memory is visible as the memory of individual processes. If the shared memory is mapped by a process, then it should be counted in the RSS / PSS. See above. The "mapping name" can sometimes be useful here (e.g.
smem -t -m).
Shmem includes files on mounted
tmpfs. You can check all mounted
df -t tmpfs.
Depending on your system,
Shmem could include some graphics buffers. I found a way to check their current size on my system (Intel graphics): Can I see the amount of memory which is allocated as GEM buffers? I would love to know if you find a different way to check on your system!
I read that
Shmem memory leaks in some other graphics driver can be associated with very large VIRT (aka VSIZE) for Xorg. Linux using whole swap, becoming unresponsive while there is plenty of free RAM
Other memory usage?
MemTotal - MemAvailable = 1796088 kB (1.8 GB)
AnonPages: 924844 kB (0.9 GB)
Shmem: 374848 kB (0.4 GB)
Of the remaining 0.5 GB, I can see a few more small uses under 0.1 GB overall. The kernel also reserves a few percentage points of margin for itself (see "low water mark"), but I guess it would be 0.2 GB or less on your system. So there is a bit more usage that I am not sure about.
Kernel slab memory
"Unreclaimable slab" memory is another terms that reduces
MemAvailable. You do not have very much:
SUnreclaim: 55044 kB (0.05 GB).
You can also run slabtop to see the list of slabs. AFAICT, slabtop doesn't give stats on slabs as reclaimable or unreclaimable. But I can usually guess, and if there is some suspicious slab I guess you can look it up by name.
3. Other tools you could use - install
atop today 8-)
smem might be overkill. Sometimes, all you need is
top, or your favourite alternative, and to know how to sort by resident memory. (Although
gnome-system-monitor might not be a good choice for this. I think it does not really show enough).
Sometimes when you have a performance problem, you need to look at disk reads & writes instead. You could use
Sometimes, you would like to have a log of per-process memory usage, so you can look at why you ran out of memory and slowed the system to a crawl...
atop is a nifty little tool that can do all the above. If this sounds useful, I suggest you install it straight away. Then you can learn about it when you need it :-).
sudo atop -R shows "PSIZE" (same meaning as "PSS"). The
atop package includes a background service that runs atop with a ten minute interval. You can use
atop -r ... to open the log files, which are stored under /var/log/atop/.