Correctly determining memory usage in Linux

Question

I'm a bit confused on some of the results I am seeing from ps and free.

On my server, this is the result of free -m

[root@server ~]# free -m
             total       used       free     shared    buffers     cached
Mem:          2048       2033         14          0         73       1398
-/+ buffers/cache:        561       1486
Swap:         2047         11       2036

My understanding of how Linux manages memory, is that it will store disk usage in RAM, so that each subsequent access is quicker. I believe this is indicated by the "cached" columns. Additionally, various buffers are stored in RAM, indicated in the "buffers" column.

So if I understand correctly, the "actual" usage is supposed to be the "used" value of "-/+ buffers/cache", or 561 in this case.

So assuming all of that is correct, the part that throws me is the results of ps aux.

My understanding of the ps results, is that the 6th column (RSS), represents the size in kilobytes the process uses for memory.

So when I run this command:

[root@server ~]# ps aux | awk '{sum+=$6} END {print sum / 1024}'
1475.52

Shouldn't the result be the "used" column of "-/+ buffers/cache" from free -m?

So, how can I properly determine the memory usage of a process in Linux? Apparently my logic is flawed.

Answer 1

Shamelessly copy/pasting my answer from serverfault just the other day :-)

The linux virtual memory system isn't quite so simple. You can't just add up all the RSS fields and get the value reported used by free. There are many reasons for this, but I'll hit a couple of the biggest ones.

• When a process forks, both the parent and the child will show with the same RSS. However linux employs copy-on-write so that both processes are really using the same memory. Only when one of the processes modifies the memory will it actually be duplicated.
  This will cause the free number to be smaller than the top RSS sum.

• The RSS value doesn't include shared memory. Because shared memory isn't owned by any one process, top doesn't include it in RSS.
  This will cause the free number to be larger than the top RSS sum.

There are many other reasons the numbers might not add up. This answer is just trying to make the point that memory management is very complex, and you cant just add/subtract individual values to get total memory usage.

Answer 2

If you're looking for memory numbers that add up have a look at smem:

smem is a tool that can give numerous reports on memory usage on Linux systems. Unlike existing tools, smem can report proportional set size (PSS), which is a more meaningful representation of the amount of memory used by libraries and applications in a virtual memory system.

Because large portions of physical memory are typically shared among multiple applications, the standard measure of memory usage known as resident set size (RSS) will significantly overestimate memory usage. PSS instead measures each application's "fair share" of each shared area to give a realistic measure.

For example here:

# smem -t
  PID User     Command                         Swap      USS      PSS      RSS
...
10593 root     /usr/lib/chromium-browser/c        0    22868    26439    49364 
11500 root     /usr/lib/chromium-browser/c        0    22612    26486    49732 
10474 browser  /usr/lib/chromium-browser/c        0    39232    43806    61560 
 7777 user     /usr/lib/thunderbird/thunde        0    89652    91118   102756 
-------------------------------------------------------------------------------
  118 4                                       40364   594228   653873  1153092 

So PSS is the interesting column here because it takes shared memory into account.
Unlike RSS it's meaningful to add it up. We get 654Mb total for userland processes here.

System-wide output tells about the rest:

# smem -tw
Area                           Used      Cache   Noncache 
firmware/hardware                 0          0          0 
kernel image                      0          0          0 
kernel dynamic memory        345784     297092      48692 
userspace memory             654056     181076     472980 
free memory                   15828      15828          0 
----------------------------------------------------------
                            1015668     493996     521672 

So 1Gb _{RAM total} = 654Mb _{userland processes} + 346Mb _{kernel mem} + 16Mb _free
(give or take a few Mb)

Overall about half of memory is used for cache (494Mb).

Bonus question: what is userland cache vs kernel cache here ?

---

btw for something visual try:

# smem  --pie=name

Answer 3

A really good tool is pmap which list the current usage of memory for a certain process:

pmap -d PID

For more information about it see the man page man pmap and also have a look at 20 Linux System Monitoring Tools Every SysAdmin Should Know, which list great tools I always use to get information about my Linux box.

Answer 4

Run top, hit h for help then f to add fields. you can add the following fields:

• RSS amount of physical memory the application is using
• CODE total amount of memory the process's executable code is using
• DATA - total amount of memory (kb) dedicated to a process's data and stack

Between these 3 you should have pretty accurate results. You can also use more detailed replacements for top I recommend htop or atop.

Edit: Almost forgot if you want really detailed information. Find the PID and cat the following file.

PID=123

cat /proc/123/status

Edit 2: If you can find it or have it the book:

Optimizing Linux Performance: A Hands-On Guide to Linux Performance Tools

-has a section Chapter 5: Performance Tools: Process-Specific Memory -it has way more information than you would ever want.

Answer 5

As others have correctly pointed out, it's hard to get a handle on actual memory used by a process, what with shared regions, and mmap'ed files and whatnot.

If you're an experimenter, you can run valgrind and massif. This may get a bit heavy for the casual user but you'll get an idea of an application's memory behavior over time. If an application malloc()'s exactly what it needs then this will give you a good representation of the real dynamic memory usage of a process. But this experiment can be "poisoned".

To complicate matters, Linux allows you to overcommit your memory. When you malloc() memory, you're stating your intent to consume memory. But allocation doesn't really happen until you write a byte into a new page of your allocated "RAM". You can prove this to yourself by writing and running a little C program like so:

// test.c
#include <malloc.h>
#include <stdio.h>
#include <unistd.h>
int main() {
    void *p;
    sleep(5)
    p = malloc(16ULL*1024*1024*1024);
    printf("p = %p\n", p);
    sleep(30);
    return 0;
}

# Shell:
cc test.c -o test && ./test &
top -p $!

Run this on a machine with less than 16GB of RAM and, voila!, you just scored 16GB of memory! (no, not really).

Notice in top you see "VIRT" as 16.004G but %MEM is 0.0

Run this again with valgrind:

# Shell:
valgrind --tool=massif ./test &
sleep 36
ms_print massif.out.$! | head -n 30

And massif says "sum of all allocs() = 16GB". So that's not very interesting.

BUT, if you run it on a sane process:

# Shell:
rm test test.o
valgrind --tool=massif cc test.c -o test &
sleep 3
ms_print massif.out.$! | head -n 30

--------------------------------------------------------------------------------
Command:            cc test.c -o test
Massif arguments:   (none)
ms_print arguments: massif.out.23988
--------------------------------------------------------------------------------


    KB
77.33^                                                                       :
     |                                                                      #:
     |                                                                :@::@:#:
     |                                                           :::::@@::@:#:
     |                                                         @:: :::@@::@:#:
     |                                                     ::::@:: :::@@::@:#:
     |                                             ::@:::@:::::@:: :::@@::@:#:
     |                                            @::@:::@:::::@:: :::@@::@:#:
     |                                            @::@:::@:::::@:: :::@@::@:#:
     |                      :@@@@@@@@@@@@@@@@@@@@:@::@:::@:::::@:: :::@@::@:#:
     |                      :@@                  :@::@:::@:::::@:: :::@@::@:#:
     |                    :@:@@                  :@::@:::@:::::@:: :::@@::@:#:
     |                    :@:@@                  :@::@:::@:::::@:: :::@@::@:#:
     |                   :@@:@@                  :@::@:::@:::::@:: :::@@::@:#:
     |                   :@@:@@                  :@::@:::@:::::@:: :::@@::@:#:
     |              :@::::@@:@@                  :@::@:::@:::::@:: :::@@::@:#:
     |          :::::@::::@@:@@                  :@::@:::@:::::@:: :::@@::@:#:
     |        :::::::@::::@@:@@                  :@::@:::@:::::@:: :::@@::@:#:
     |       ::::::::@::::@@:@@                  :@::@:::@:::::@:: :::@@::@:#:
     |       ::::::::@::::@@:@@                  :@::@:::@:::::@:: :::@@::@:#:
   0 +----------------------------------------------------------------------->Mi
     0                                                                   1.140

And here we see (very empirically and with very high confidence) that the compiler allocated 77KB of heap.

Why try so hard to get just heap usage? Because all of the shared objects and text sections that a process uses (in this example, the compiler) are not terribly interesting. They're constant overhead for a process. In fact, subsequent invocations of the process almost come for "free".

Also, compare and contrast the following:

MMAP() a 1GB file. Your VMSize will be 1+GB. But you're Resident Set Size will only be the portions of the file that you caused to be paged in (by dereferencing a pointer to that region). And if you "read" the whole file then, by the time you get to the end, the kernel may have already paged out the beginnings (this is easy to do because the kernel knows exactly how/where to replace those pages if dereferenced again). In either case, neither VMSize nor RSS are a good indicator of your memory "usage". You haven't actually malloc()'ed anything.

By contrast, Malloc() and touch LOTS of memory -- until your memory gets swapped to disk. So your allocated memory now exceeds your RSS. Here, your VMSize might start to tell you something (your process owns more memory than what actually resides in your RAM). But it's still difficult to differentiate between VM that is shared pages and VM that is swapped data.

This is where valgrind/massif gets interesting. It shows you what you've intentionally allocated (regardless of the state of your pages).

Answer 6

ps gives you the amount of memory used by each process. Some of that memory is mmapped files, which counts under cache. Some of that memory (especially code) is shared with other processes, so if you add up the RSS values it's counted multiple times.

There's no right answer to “how much memory does this process use?”, because it doesn't depend on the process alone, it also depends on the environment. There are many different values you might call the “memory usage” of the process, and they don't match or add up because they're counting different things.

Answer 7

Try this: it will give you the total RAM actually used by all the process running in MB

ps -eo size,pid,user,command --sort -size | awk '
  { hr=$1/1024 ; printf("%13.2f Mb ",hr) } 
  { for ( x=4 ; x<=NF ; x++ ) { printf("%s ",$x) } print "" }
  ' | awk '{total=total + $1} END {print total}'

Answer 8

It ll show you how much memory user by users..

#!/bin/bash
total_mem=0

printf "%-10s%-10s\n" User MemUsage

while read u m
do
        [[ $old_user != $u ]] && {  printf "%-10s%-0.1f\n" $old_user $total_mem;
                                    total_mem=0; }
        total_mem="$(echo $m + $total_mem | bc)"
        old_user=$u

done < <(ps --no-headers -eo user,%mem| sort -k1)

#EOF

Answer 9

Use this command to find memory utilization in %.

Memory utilised:

grep Mem | awk '{print $3/$2 * 100.0}'

free memory

grep Mem | awk '{print $4/$2 * 100.0}'