Linux (RHEL) problem with cached and swap memory

Question

I have a problem with linux cached and swap memory. I know that linux is optimising itself when reading data from the disk (buffering it in the RAM) but in my case it seems that this part of memory is causing apps to use swap space and because of that the apps start to work really slow (my issue is with java server running on that linux). Here is the output:

[glassfish@pahod001 logs]$ free -m
             total       used       free     shared    buffers     cached
Mem:         32108      31031       1076          1        452      14980
-/+ buffers/cache:      15598      16509
Swap:         8191         49       8142

So now it seems that everything is almost OK. Even with cached included we still have free of 1076MB RAM. But after some time (I count time from the java server restart -> about 2 days) java process is starting to use swap space (can be seen in /proc/<proc id>/status). Today morning it started using 10MB of RAM already, when cached data is almost 15GB. Why linux is prioritizing cached memory over what my application really needs and moving it into the swap? Should I disable swap? System uptime is 737 days (it is important server), maybe there is something stuck in cached and should be cleaned manually? It is production server, so everything must be done with caution.

Process restarted 2 days ago is already using swap, when there is 15,5GB free RAM. I cannot understand this cached memory management style...

[glassfish@pahod001 logs]$ cat /proc/20122/status
Name:   java
State:  S (sleeping)
Tgid:   20122
Pid:    20122
PPid:   1
TracerPid:      0
Uid:    537     537     537     537
Gid:    537     537     537     537
Utrace: 0
FDSize: 256
Groups: 537
VmPeak:  5995004 kB
VmSize:  5995000 kB
VmLck:         0 kB
VmHWM:   1017892 kB
VmRSS:    759988 kB
VmData:  5843144 kB
VmStk:        88 kB
VmExe:         4 kB
VmLib:     16624 kB
VmPTE:      2600 kB
VmSwap:     2844 kB

Answer 1

Your Java is hogging 6 GB (almost all of it data), and there is 3 MB (0.0005%) of swap that worries you? How about the 46MB used swap that is not for that process -- what is using that, and how often?

Sure, active processes should not be swapped out in favour of cache. But that 3 MB of Java might not have been used since it loaded, several days ago. You don't know it is actually swapping anything, just that the algorithm for swapping out decides it could use RAM better.

Can you account for why Java needs to hold onto 6 GB forever? Seems to me it may have a memory leak, or an algorithm that has unacceptable behaviour with large data.

I think you need tools that give you a broader diagnostic view of the whole system, but I don't have RHEL on hand.