Question: how can we report the REAL memory usage (without the cache!) using nmon or vmstat or svmon on AIX 6?

enter image description here

enter image description here

enter image description here

Like on Linux, we can use the free command, but it's not available in AIX:

[user@notebook ~]$ free -m
         total       used       free     shared    buffers     cached
Mem:          7797       4344       3453          0        219       2745
-/+ buffers/cache:       1379       6417
Swap:         2047          0       2047
[user@notebook ~]$ free -m | grep cache: | awk '{print $3}'
[user@notebook ~]$ 

Short version: look at the in use clnt+pers pages in the svmon -G output (unit is 4k pages) if you want to know all file cache, or look at vmstat -v and look at "file pages" for file cache excluding executables (same unit).

You should check out the following article if you want a good overview of what's going on: Overview of AIX page replacement.

For an extremely short summary, memory in AIX is classified in two ways:

  • Working memory vs permanent memory

    • Working memory is process (stack, heap, shared memory) and kernel memory. If that sort of memory needs to be pages out, it goes to swap.

    • Permanent memory is file cache. If that needs to be paged out, it goes back out to the filesystem where it came from (for dirty pages, clean pages just get recycled). This is subdivided into non-client (or persistent) pages for JFS filesystems, and client pages for JFS2, NFS, and possibly others.

  • Computational vs non-computational pages.

    • Computational pages are again process and kernel data, plus process text data (i.e. pages that cache the executable/code).

    • Non-computational are the other ones: file cache that's not executable (or shared library).

svmon -G (btw, svmon -G -O unit=MB is a bit friendlier) gives you the work versus permanent pages. The work column is, well, work memory. You get the permanent memory by adding up the pers (JFS) and clnt (JFS2) columns.

In your case, you've got about 730MB of permanent pages, that are backed by your filesystems (186151*4k pages).

Now the topas top-right "widget" FileSystemCache (numperm) shows something slightly different, and you'd get that same data with vmstat -v: that's only non-computational permanent pages. i.e. same thing as above, but excluding pages for executables.

In your case, that's about 350MB (2.2% of 16G).

Either way, that's really not much cache.

  • please also note filesystem cache is not really 'in use memory', and depending on the minperm% the fscache gets freed when the memory consumption gets too high. – Mike van Hoof Oct 20 '15 at 9:23
  • So, if I am understanding this correctly. Suppose topas gives the values below. Then only 51% of non-cache memory is being used? Also, how can the percentages of real memory exceed 100%? MEMORY Real,MB 31744 % Comp 51 % Noncomp 39 % Client 39 – Timothy Pulliam Aug 27 '18 at 18:55
  • @TimothyPulliam: the client part doesn't add. see the "permanent" bullet point above. – Mat Aug 27 '18 at 19:08

The command you are looking for (imho) is:

# svmon -P -O filtertype=working,segment=off,filtercat=exclusive,unit=MB

The key options here are:

  • 'filtertype=working' # aka, no cache;
  • 'segment=off' # actually default when using -O
  • 'filtercat=exclusive' # do not include shared memory or kernel atm
  • 'unit=MB' # who want to calculate from # of pages ??

And, you will want to look at other options such as -C (command name related, some examples below) and perhaps -U (user related)

++++ start of comment ++++

inserting what I would have entered as a comment to your question, but lack the reputation - as a new user here.

Your vmstat output tells me more than just your current situation - as it is the single line output - it is historical - and I suspect you have been having memory issues as it shows a history of pi/po (paging space page in/ paging space pageout)

Other columns of interest are the fr/sr columns:

  • fr: pages freed by lrud (least recently used daemon aka page stealer)
  • sr: pages scanned/searched by lrud looking for an 'old' page
  • sr/fr: ratio that expresses how many pages must be "scanned" to free 1

What I consider troubling are the pi/po values given here - and completely out of line with the data from the other commands - also no uptime here, so hard to know what for 'test' generated these numbers.

  • pi: paging space page in (i.e., read application memory from paging space)
  • po: steal memory and write application (aka working) memory to paging space - only working memory goes to/from page space

In your presentation you show pi=22 and po=7. This means, on average, the system was reading information from paging space (after it had been written) 3x more often than it wrote data. This is an indication of a starved system because data is being read-in (pi) then stolen again (sr/fr) before it is ever touched (referenced aka used) - or readin and removed again before the application 'waiting' for it ever has a chance to access it.

In short, the data presented is not 'in sync' with the 'pain' moments - although it might explain why only 2.2% of your memory is now used for caching (it may even be 'computational aka the loaded programs').

As far as vmstat goes I also suggest the flags -I (capital:i which adds 'fi' and 'fo' - fileIn and fileOut activity) and -w (wide) so the numbers are better positioned under the textual headers.

++++ end of 'comment'

So let's see an excerpt using -P (process view)

# svmon -P -O filtertype=working,segment=off,filtercat=exclusive,unit=MB | head -15 

    Unit: MB

     Pid Command          Inuse      Pin     Pgsp  Virtual
14614630 httpd             21.5     0.06        0     21.5
11272246 httpd             21.4     0.06        0     21.4
12779758 httpd             21.2     0.06        0     21.2
17760476 httpd             20.9     0.06        0     20.9
11796712 httpd             20.8     0.06        0     20.8
17039454 httpd             20.6     0.06        0     20.6
11862240 httpd             20.6     0.06        0     20.6
14680090 httpd             20.5     0.06        0     20.5
10747970 httpd             20.5     0.06        0     20.5
11141286 httpd             20.5     0.06        0     20.5
 4718766 mysqld            13.6     0.02        0     13.6

When you are not root you only see the commands in your environment.

$ svmon -P -O filtertype=working,segment=off,filtercat=exclusive,unit=MB
Unit: MB

     Pid Command          Inuse      Pin     Pgsp  Virtual
 5505172 svmon             10.7     0.19     0.44     11.4
 6553826 ksh               0.57     0.02        0     0.57
 9175288 ksh               0.55     0.02        0     0.55
12910710 sshd              0.55     0.02        0     0.55
15204356 sshd              0.52     0.02        0     0.52
12779760 head              0.18     0.02        0     0.18

You may want to look at a specific command - so switching back to root to look at httpd


svmon -C httpd -O filtertype=working,segment=off,filtercat=exclusive,unit=MB 
Unit: MB
Command                              Inuse      Pin     Pgsp  Virtual
httpd                               227.44     0.69        0   227.44

Details: excerpt

    # svmon -C httpd -O filtertype=working,segment=category,filtercat=exclusive,unit=MB >
Unit: MB
Command                              Inuse      Pin     Pgsp  Virtual
httpd                               230.62     0.81        0   230.62

EXCLUSIVE segments                   Inuse      Pin     Pgsp  Virtual
                                    230.62     0.81        0   230.62

    Vsid      Esid Type Description              PSize  Inuse   Pin Pgsp Virtual
  81a203         3 work working storage              m   24.6     0    0    24.6
  8b82d7         3 work working storage              m   18.8     0    0    18.8
  8b9d37         3 work working storage              m   18.2     0    0    18.2
  8915f2         f work shared library data          m   2.00     0    0    2.00
  89abb3         f work shared library data          m   2.00     0    0    2.00
  824ea4         f work shared library data          m   2.00     0    0    2.00

This does not show off the 'segment=category' well, so now with a simpler command - tail - and show a summary and detail of each memory 'segment' type - but still 'working' memory only (aka no caching)

# svmon -C tail -O filtertype=working,segment=category,unit=MB                    
Unit: MB
Command                              Inuse      Pin     Pgsp  Virtual
tail                                  82.5     52.6     5.12     90.6

SYSTEM segments                      Inuse      Pin     Pgsp  Virtual
                                      34.1     33.1     2.38     35.8

    Vsid      Esid Type Description              PSize  Inuse   Pin Pgsp Virtual
   10002         0 work kernel segment               m   34.1  33.1 2.38    35.8

EXCLUSIVE segments                   Inuse      Pin     Pgsp  Virtual
                                      0.18     0.02        0     0.18

    Vsid      Esid Type Description              PSize  Inuse   Pin Pgsp Virtual
  88b4f1         f work working storage             sm   0.09     0    0    0.09
  82d005         2 work process private             sm   0.07  0.02    0    0.07
  8e0c9c         3 work working storage             sm   0.02     0    0    0.02

SHARED segments                      Inuse      Pin     Pgsp  Virtual
                                      48.2     19.5     2.75     54.6

    Vsid      Esid Type Description              PSize  Inuse   Pin Pgsp Virtual
    9000         d work shared library text          m   48.2  19.5 2.75    54.6

nmon then hit "m" will quickly show you a few big uses of memory

ipcs -am

The shared memory used by lots of applications like DB2 and Oracle - check the SEGSZ for the size from the ipcs -am command. The Owner column usually tells you what it is used for like the oracle user for the SGA or db2inst1 for the DB2 buffer cache.

Then it is down to the processes and this gets tricky. All processes running the same program file will be sharing the code pages as its read-only. The may also be sharing some or nearly all of the data and sack pages if the processes started by a common process that then forked like for example RDBMS and things like Apache. This is also true for the dozens of libraries that processes also need and are largely invisible to us.

Due to this unknown sharing it is often the case if you add up all the memory of all the processes it is obviously much larger than memory.

If you use nmon then "t" for top processes and then "4" to order in process size you see the process memory.

  • Size KB = the size as found in the program file on disk.
  • Resident Set Size = how big it is in memory (excluding the pages still in the file system (like code) and some parts on paging disks)".
  • ResText column is the code pages of the Resident Set
  • ResData column is the data and stack pages of the Resident Set

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.