What is "System Memory " displayed in nmon in AIX?

Question

I have a host with AIX 6.1 (6100-04-02-1007) upgraded from AIX 5.1 with Veritas Storage Foundation 5.1.

Since the upgrade from AIX 5.3 to 6.1, Run vx services means a lots of memory consumed that I can not explain. This memory appears in 'nmon' as System (see below System 41.2%). If I do not start Veritas it is <20%.

Whay does mean that "System" memory? How can I know what is consuming this memory?

 Memory ───────────────────────────────────────────────────────────────────────
          Physical  PageSpace |        pages/sec  In     Out | FileSystemCache 
% Used       93.9%     10.7%  | to Paging Space   0.0    0.0 | (numperm)  3.4% 
% Free        6.1%     89.3%  | to File System    6.0    0.0 | Process   49.3% 
MB Used    7695.0MB   871.5MB | Page Scans    20430.7        | System    41.2% 
MB Free     497.0MB  7304.5MB | Page Cycles       1.5        | Free       6.1% 
Total(MB)  8192.0MB  8176.0MB | Page Steals       0.0        |           ------
                              | Page Faults   10066.8        | Total    100.0% 
------------------------------------------------------------ | numclient  3.4% 
Min/Maxperm     237MB(  3%)  7105MB( 87%) <--% of RAM        | maxclient 86.7% 
Min/Maxfree     960   1088       Total Virtual   16.0GB      | User      49.7% 
Min/Maxpgahead    2      8    Accessed Virtual    7.5GB 47.2%| Pinned    41.3% 

Answer 1

Anything like this should be in the man page

In particular:

Numperm     Number of frames used for files (in 4 KB pages).  
Process     Percentage of real memory used by process segments.  
System      Percentage of real memory used by system segments.  
Free        Percentage of real memory that is free.

Answer 2

You can find some information on AIX memory segments here: http://www-01.ibm.com/support/docview.wss?uid=swg1IY99425.

As for the use of memory by different subsystems I can recommend this article: https://www.ibm.com/developerworks/wikis/display/WikiPtype/AIXmemuse

Answer 3

I am pretty sure that this memory is being used by Veritas Modules, but: how can I check that? is it possible?

The traditional way to find out what processes are using memory is to use the ps command.

ps -e -o user,pid,shmpgsz,spgsz,tpgsz,vmsize,vsz,args

Answer 4

The numbers you list under system are for FileSystem Cache values, and you can tune those as needed to reduce the amount of memory that will be allocated to cache the filesystem.

You'll want to focus on these parameters, reducing the allocation slowly until you get to a level you find workable for your situation:

lru_file_repage=0
minperm% = 5
maxclient% = 55
maxperm% = 60

Better would be to get into the veritas suggested settings, since if your storage manager already handles caching, the OS might be told to not cache at all (or very lightly) so that you can let the storage software do it's thing and the OS focus on running it's workload that exists outside the verities environment.