I've been monitoring a Java process (Oracle Weblogic) using TOP and have observed that sometimes the following will happen:
- RES will increase from it's usual stable level of 2,097,152 KB up to almost 10GB over the course of an hour or so, then drop to zero, then revert to 2,097,152 KB
- Through this entire time, VIRT memory remains constant at 6,904,832KB
- %MEM increases while RES is increasing, and keeps increasing when RES drops (system has 32GB of RAM)
I have read a number of explanations of what RES, VIRT and %MEM are, including man top, and this doesn't really fit with any of them. Specifically, I thought RES and %MEM must be completely correlated, and that RES can never be higher than VIRT.
Can anyone explain what is going on?
I've included an image below showing behaviour over time.
It may be relevant that this is one node of load-balanced system, and the other node carries on as though nothing has happened. I have seen this half a dozen times on either node across different days. It does cause a short, small spike in end-user response times, at the point where RES drops. Both nodes are very close to identical. The admin has set various memory flags have been set on the Java process. Verbose GC is turned on, and there was no full GC during the period below.