4

I have some doubts with memory usage.
Currently I have nagios check who measures used memory from free -m command on Linux. I added another script for memory usage from http://exchange.nagios.org/directory/Plugins/Operating-Systems/Solaris/check_mem-2Epl/details and it measures for Solaris from vmstat and for Linux from /proc/meminfo, with this new check a have a lot more usage 20 % - 30 % more on some hosts.

./check_mem.pl -f -w 90 -c 60
CRITICAL - 34.6% (439872 kB) free!|TOTAL=1272376KB;;;; USED=832504KB;127237;508950;; FREE=439872KB;;;; CACHES=418977KB;;;;

this is output from new script on Solaris host on this host I also have 

    vmstat 1 2
 kthr      memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr s0 -- -- --   in   sy   cs us sy id
 0 0 0 1184172 474856 54 222 0  0  0  0 112 8  0  0  0  231 1735  669  1  8 91
 0 0 0 1175352 440948 16 58  0  0  0  0  0  0  0  0  0  229   83  190  0  3 97

values from vmstat and from new script are OK. I mean the script collects from vmstat values ok. I need to know what is the best way to measure memory usage on Solaris and Linux, how can I see how much OS uses and how much is used by app is it vmstat(Sunos) and /proc/meminfo (Linux) OK for that?

Improve this question
8

vmstat is not that useful to measure memory usage. It doesn't give any metric quantifying how much virtual and physical memory is used and what is using it. It is however a very good tool to measure RAM shortage. You just need to monitor the sr column (scan rate). As long as it stays equal to zero, you shouldn't worry about RAM. If it is not equal to zero, you should investigate what is demanding RAM.

To have a detailed idea about what is using your system memory, in addition to the already suggested echo ::memstat | mdb -k command, you can run prstat -n 1 -a which will give you the memory usage per user, prstat -n 1 -Z for zone usage and prstat -s rss for per process usage sorted by RAM.

In prstat output, the SWAP column shows the virtual memory used and the RSS column, the RAM used.

About kernel usage, you can run kstat -n system_pages and have a look to the pp_kernel value.

To precisely answer to your question:

how can I see how much OS uses and how much is used by app is it vmstat(Sunos) and /proc/meminfo (Linux) OK for that?

No vmstat is not OK for that on Solaris. You might run echo ::memstat | mdb -k and parse its output.

The sum of the lines "Anon" and "Exec and libs" is the RAM used by applications, The line "Kernel" reports the RAM used by the kernel, the lines "ZFS ...", "Page cache" and "Free (cachelist)" show RAM used to cache data and the line "Free (freelist)" reports unused, i.e. wasted RAM.

Note: all of the above is about Solaris.

Improve this answer
5
  • thaks for that, also as i understood real usage i get from vmstat when i dont care who actualy uses, what is what i actualy want ? – klerk Jan 21 '14 at 8:36
  • @vladeli Sorry, I don't understand what you wrote. – jlliagre Jan 21 '14 at 14:37
  • ok so question just for solaris i find out for Linux, who to belive more prstat or vmstat, with prstat i can parse root, system and default values with prstat -J 1 1, so what is generaly used memory difference between vmstat and prstat, as i saw prstat can divide used memory by users? – klerk Jan 23 '14 at 11:32
  • You can believe both prstat and vmstat output. They just happen to report different things. Which are the ones you are looking for is still unclear. – jlliagre Jan 23 '14 at 11:38
  • well i looking for used memory without OS usage, as i sad i get different values with prstat and vmstat. With vmstat i get more used memory then prstat, so my question is how to calculate TOTAL USED memory – klerk Jan 23 '14 at 11:41
2

Yes, vmstat and /proc/meminfo are the best options On Solaris you can also use following command: 

echo ::memstat | mdb –k

It is really slow (it can take up to 30minutes on some bigger systems:)), but will give you a nice overview of the memory usage, hot much is used by kernel, processes, ZFS etc.

Improve this answer
3
  • 1
    ::memstat taking a excessive time was a bug that should be fixed on current system. The patch was released several years ago and is included with Solaris 10 update 8 (Nov 2010) and newer. – jlliagre Jan 20 '14 at 21:47
  • CRITICAL - 34.6% (439872 kB) free!|TOTAL=1272376KB;;;; USED=832504KB;127237;508950;; FREE=439872KB;;;; CACHES=418977KB;;;; – klerk Jan 20 '14 at 21:59
  • this output i get whit vmstat and prtconf $used_memory_kb = $total_memory_kb - $free_memory_kb, $total_memory i parse with prtconf and free_momory i parse with vmstat 1 2 – klerk Jan 20 '14 at 22:03
0

The "best way" is not necessarily to distinguish betweem how much OS uses and how much is used by app, as you imply.

vmstat 1 2

This command gives a per second value, which is an important aspect. You can use -Sm to prevent layout mistakes due to long numbers.

]# vmstat 1 3
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2  0      0 6291912  22108 991804    0    0     2    24    7   29  7  3 90  0  0
 0  0      0 6291156  22108 992488    0    0     0     0   79  251  0  0 100  0  0
 0  0      0 6291156  22116 992568    0    0     0    44   51   85  0  0 100  0  0
]# vmstat 1 3 -Sm
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2  0      0   6442     22   1016    0    0     2    24    7   29  7  3 90  0  0
 0  0      0   6442     22   1015    0    0     0     0   73  212  0  0 100  0  0
 0  0      0   6442     22   1015    0    0     0     0   87  186  0  0 100  0  0

Here, just like in free, you can answer the basic questions:

1) Free Memory

This is a mm value; the command free uses it to calculate "used" from "total". Total physical RAM is not shown above. The -s option gives some additional values:

]# vmstat -s -Sm |grep total
         8254 m total memory
            0 m total swap

2) Swap

The output of vmstat is very different between your Solaris example six years ago and Linux now. swpd is an interesting field, It gives some scan activity of kswapd. The si and so fields show the activity. By choosing a good delay and count you see exactly what is going on. Size and use/free of swap is in free (and /proc/meminfo).

3) Cache ("pagecache", ...)

This is where it gets complicated in detail. vmstat -a shows active and inactive instead of buff and cache, like free, as buff/cache.

I did a echo 1 > /proc/sys/vm/drop_caches yesterday, even though it hurt. Now I did grep -r ... at the top of a linux 5 kernel source and monitored with vmstat 7 100 -Sm: 

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0   6361     27   1069    0    0     0     0   76  159  0  0 100  0  0
 0  0      0   6395     27   1068    0    0     0    13  263  419  0  0 100  0  0
 1  0      0   6278     30   1185    0    0 15767     0 1863 3680  1  2 96  2  0
 2  0      0   6128     33   1332    0    0 20570     0 2716 5158  2  4 92  2  0
 1  0      0   5940     38   1513    0    0 25500     0 2456 4645  2  4 92  2  0
 0  1      0   5770     40   1682    0    0 23369     0 2064 3784  2  4 92  2  0
 1  0      0   5472     43   1978    0    0 41068     3 1960 3484  3  3 91  3  0
 1  0      0   5367     44   2079    0    0 14870     0 1139 2118  2  2 95  1  0
 0  0      0   5367     44   2079    0    0     0   201   39   96  0  0 100  0  0
 0  0      0   5367     44   2079    0    0     0     0   37   95  0  0 100  0  0
 0  0      0   5370     44   2079    0    0     0     0   99  203  0  0 100  0  0
 0  0      0   5370     44   2079    0    0     0     0  248  446  0  0 100  0  0

So the 1 GB of the source files got (re)loaded into the Cache, via block device. CPU was only marginally touched.

The importance is: free went down, but becasue it was only used for cache, it is readily available.

There is also another kind of cache, which is a kernel slab cache. 

/proc/meminfo:KReclaimable:       601200 kB
/proc/meminfo:SReclaimable:       601200 kB
/proc/meminfo:SUnreclaim:          50048 kB
/proc/vmstat:nr_slab_reclaimable  150300 
/proc/vmstat:nr_slab_unreclaimable 12512

In the proc/vmstat file it is pages, not kB, so multiply by 4. SReclaimable is mentioned in man free.

4) Details

After free and vmstat, vmstat -s and vmstat DELAY COUNT, with their fields and terminology, there is /proc/vmstat file. It contains a lot of additional values.

Just for completeness: top gives a per process memory usage. But that is another story.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.