I'm trying to figure out why memory consumption started to constantly increase on my server during the last hours. I've tried to find the cause on the application level, but no success. That is why now looking into possible server cause. I'm not a pro in servers administration, so any help is appreciated. First common memory was eaten, now swap consumption is also constantly increasing.
My server runs on CentOS 7 with the kernel 3.10.0-514.26.2.el7.x86_64
Finally, the issue was identified to be caused by a recently updated server library. The accepted answer is a good reminder, in the situation when you're stressed out by the memory usage, to trace back what had been changed in your system before the issue appeared.
Some tips I've been looking for and found to be very useful are described in How to find which processes are taking all the memory?
I'm listing below the commands that I used and may help in such situation.
ps auwx --sort rss - processes sorted by memory usage
ps -fu username - processes by a user
htop usage/analysis showed many hung application cron-launched processes in my case. I configured htop to output both PID and PPID, because I needed to correlate PPID to /var/log/cron logged processes.
total used free shared buff/cache available Mem: 7565 6525 440 47 599 657 Swap: 8191 2612 5579
MemTotal: 7747260 kB MemFree: 253960 kB MemAvailable: 498904 kB Buffers: 6160 kB Cached: 189076 kB SwapCached: 467788 kB Active: 5572588 kB Inactive: 1258540 kB Active(anon): 5498664 kB Inactive(anon): 1185908 kB Active(file): 73924 kB Inactive(file): 72632 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 8388604 kB SwapFree: 5686452 kB Dirty: 104 kB Writeback: 0 kB AnonPages: 6168400 kB Mapped: 68668 kB Shmem: 48676 kB Slab: 456672 kB SReclaimable: 389064 kB SUnreclaim: 67608 kB KernelStack: 7232 kB PageTables: 106848 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 12262232 kB Committed_AS: 10244216 kB VmallocTotal: 34359738367 kB VmallocUsed: 26276 kB VmallocChunk: 34359705340 kB HardwareCorrupted: 0 kB AnonHugePages: 5191680 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 83968 kB DirectMap2M: 8300544 kB
How can I proceed to find if there's any issue in how the server is functioning or configured in terms of memory usage?