The "best way" is not necessarily to distinguish betweem how much OS uses and how much is used by app, as you imply.
vmstat 1 2
This command gives a per second value, which is an important aspect. You can use -Sm
to prevent layout mistakes due to long numbers.
]# vmstat 1 3
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 0 0 6291912 22108 991804 0 0 2 24 7 29 7 3 90 0 0
0 0 0 6291156 22108 992488 0 0 0 0 79 251 0 0 100 0 0
0 0 0 6291156 22116 992568 0 0 0 44 51 85 0 0 100 0 0
]# vmstat 1 3 -Sm
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 0 0 6442 22 1016 0 0 2 24 7 29 7 3 90 0 0
0 0 0 6442 22 1015 0 0 0 0 73 212 0 0 100 0 0
0 0 0 6442 22 1015 0 0 0 0 87 186 0 0 100 0 0
Here, just like in free
, you can answer the basic questions:
1) Free Memory
This is a mm value; the command free
uses it to calculate "used" from "total". Total physical RAM is not shown above. The -s
option gives some additional values:
]# vmstat -s -Sm |grep total
8254 m total memory
0 m total swap
2) Swap
The output of vmstat is very different between your Solaris example six years ago and Linux now. swpd
is an interesting field, It gives some scan activity of kswapd
. The si and so fields show the activity. By choosing a good delay and count you see exactly what is going on. Size and use/free of swap is in free
(and /proc/meminfo).
3) Cache ("pagecache", ...)
This is where it gets complicated in detail. vmstat -a
shows active and inactive instead of buff and cache, like free
, as buff/cache.
I did a echo 1 > /proc/sys/vm/drop_caches
yesterday, even though it hurt. Now I did grep -r ...
at the top of a linux 5 kernel source and monitored with vmstat 7 100 -Sm
:
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 0 6361 27 1069 0 0 0 0 76 159 0 0 100 0 0
0 0 0 6395 27 1068 0 0 0 13 263 419 0 0 100 0 0
1 0 0 6278 30 1185 0 0 15767 0 1863 3680 1 2 96 2 0
2 0 0 6128 33 1332 0 0 20570 0 2716 5158 2 4 92 2 0
1 0 0 5940 38 1513 0 0 25500 0 2456 4645 2 4 92 2 0
0 1 0 5770 40 1682 0 0 23369 0 2064 3784 2 4 92 2 0
1 0 0 5472 43 1978 0 0 41068 3 1960 3484 3 3 91 3 0
1 0 0 5367 44 2079 0 0 14870 0 1139 2118 2 2 95 1 0
0 0 0 5367 44 2079 0 0 0 201 39 96 0 0 100 0 0
0 0 0 5367 44 2079 0 0 0 0 37 95 0 0 100 0 0
0 0 0 5370 44 2079 0 0 0 0 99 203 0 0 100 0 0
0 0 0 5370 44 2079 0 0 0 0 248 446 0 0 100 0 0
So the 1 GB of the source files got (re)loaded into the Cache, via block device. CPU was only marginally touched.
The importance is: free
went down, but becasue it was only used for cache, it is readily available.
There is also another kind of cache, which is a kernel slab cache.
/proc/meminfo:KReclaimable: 601200 kB
/proc/meminfo:SReclaimable: 601200 kB
/proc/meminfo:SUnreclaim: 50048 kB
/proc/vmstat:nr_slab_reclaimable 150300
/proc/vmstat:nr_slab_unreclaimable 12512
In the proc/vmstat file it is pages, not kB, so multiply by 4. SReclaimable
is mentioned in man free.
4) Details
After free
and vmstat
, vmstat -s
and vmstat DELAY COUNT
, with their fields and terminology, there is /proc/vmstat file. It contains a lot of additional values.
Just for completeness: top
gives a per process memory usage. But that is another story.