Linux memory usage?

Question

We have 1TB of memory on a server which hosts an SAP application.

When the app is running the memory usage shown in top is around 700GB. When the app is stopped the memory usage shown in top comes down to 10GB. A reboot of the server gets the memory usage to 1GB.

• Even though no application is running, why does top show 10GB used and gains 9GB after reboot?
• Is it possible to gain that 9GB without a reboot?

Output of free -g:

free -g 
                 total used   free shared buffers cached 
Mem:              1009  567   442    0      0     152 
-/+ buffers/cache:      415   594 
Swap:              1     0     1

Answer 1

Linux uses RAM in a different way from what other operating systems do.

Rather than sitting there with unused RAM, Linux stores data that it thinks might be used in RAM-any applications may be cached here, files, etc.

As a result, Linux RAM usage is higher than what is used by running applications. This extra usage is buffered to be sued by other things. Run free -h and the second row under used will show you that a lot of the "used" memory is really just cached.

In the case where all of your memory is cached and a program needs memory, it will remove enough from the cache to fit that program.

Answer 2

It is due to some file descriptors still open though the app is stopped. You can list the open file descriptors using the techniques mentioned here.

If you need to close the file descriptors without rebooting, you can follow the approach mentioned by Graeme here.However, you need to be aware of the file descriptors that you are closing as highlighted by Graeme in his answer. His answer is,

To answer literally, to close all open file descriptors for bash:

for fd in $(ls /proc/$$/fd); do
  eval "exec $fd>&-"
done

However this really isn't a good idea since it will close the basic file descriptors the shell needs for input and output. If you do this, none of the programs you run will have their output displayed on the terminal (unless they write to the tty device directly). If fact in my tests closing stdin (exec 0>&-) just causes an interactive shell to exit.

What you may actually be looking to do is rather to close all file descriptors that are not part of the shell's basic operation. These are 0 for stdin, 1 for stdout and 2 for stderr. On top this some shells also seem to have other file descriptors open by default. In bash you have 255 (also for terminal I/O) and dash I have 10 which points to /dev/tty rather than the specific tty/pts device the terminal is using. To close everything apart from 0, 1, 2 and 255 in bash:

for fd in $(ls /proc/$$/fd); do
  case "$fd" in
    0|1|2|255)
      ;;
    *)
      eval "exec $fd>&-"
      ;;
  esac
done

Note also that eval is required when redirecting the file descriptor contained in a variable, if not bash will expand the variable but consider it part of the command (in this case it would try to exec the command 0 or 1 or whichever file descriptor you are trying to close). Also using a glob instead of ls (eg /proc/$$/fd/*) seems to open an extra file descriptor for the glob, so ls seems the best solution here.

Update

For further information on the portability of /proc/$$/fd, please see Portability of file descriptor links. If /proc/$$/fd is unavailable, then a drop in replacement for the $(ls /proc/$$/fd), using lsof (if that is available) would be $(lsof -p $$ -Ff | grep f[0-9] | cut -c 2-).