This is an old post, however, I would still take the liberty of putting up my thoughts here.
Starting from down under, Linux would first divide the memory into pages (usually 4K per page on x86_64 system). Thereafter, virtual memory is created, whose mapping is done with physical memory using MMU (Memory Management Unit).
Processes are allocated memory from the virtual memory area, so please note, when you see /proc/meminfo, you will see VMalloc* as the virtual memory details.
Lets say you have a process which requests for memory (say 300MB - a web browser). The process would be allocated 300MB from the virtual memory,
however, it is not necessary it is memory mapped (that is mapped to physical memory). There is concept of "Copy on Write" for memory management, whereby, if your processes actually uses the memory allocated from virtual memory (that is it does some write on the memory), only then it is mapped to physical memory. This assists the kernel to work properly in a multi-process environment efficiently.
What are cache?
A lot of memory used by processes are shared. Lets say the glibc library is used by almost all processes. What is the point of
keeping multiple copies of glibc in the memory, when every process could access same memory location and do the job. Such frequently
used resources are kept in cache so that when processes demand, they could be referenced to same memory location. This helps in speeding up
processes, as reading glibc(etc.) again & again from disk would be time consuming.
The above was for shared libraries per say, similar is also true to file reading as well. If you read a large file (say 100-200MB) for the first time,
it would take a lot of time. However, when you try and do the same read again, it would be faster. Data was cached in memory, and re-read was not done for all blocks.
What is buffer?
As far as buffer is concerned, when a processes does file I/O, it relies on kernel's buffer to write data to disk. The processes, requests
the kernel to do the job. So, on behalf of the process, the kernel writes the data to its "buffer", and tells process that the write is done.
In an async manner, kernel will keep syncing this data in buffer to disk. In this way, the processes is relying on the kernel to choose a
correct time to sync data to disk, and the processes could continue working ahead. Remember, this is general I/O that normal processes are doing.
However, specialized processes, which need to confirm that I/O is actually done on the disk can use other mechanism to do I/O on disk.
Some of opensource utilities are libaio. Also, there are ways to call explicit sync to FDs opened in your processes context, so that you force the
kernel the kernel to sync the data to disk for the write you might have done.
What are page faults then?
Consider an example, when you start a process (say a web browser), whose binary is about 300MB. However, the complete 300MB of the web browser binary
does not start working instantly. The process keeps moving from functions-to-functions in its code. As said earlier, Virtual Memory would be 300MB consumed
however, not all is memory mapped to physical memory (RSS - resident memory would be less, see top output). When code execution reaches a point, for which memory is not actually physically mapped, a page
fault would be issues. Kernel would map this memory to physical, associate the memory page to your process. Such a page fault is called "Minor Page
Similarly speaking, when a process is doing file I/O major page faults are raised.
When and why Swap Out happens?
Inline with the details above, lets consider a scenario when the good amount of memory becomes memory mapped. And now a processes
starts up, which requires memory. As discussed above, kernel will have do some memory mapping. However, there not enough physical RAM
available to map the memory. Now, the kernel will first look into the cache, it will have some old memory pages which are not being used.
It will flush those pages onto a separate partition (called SWAP), free up some pages, and map freed pages to the new request coming.
As disk write is much slower than solid-state RAM, this process takes a lot of time, and hence a slow down is seen.
Lets say you see a lot of free memory available in the system. Even then you see that there is a lot of swap-out happening.
There could be a probable issue of memory fragmentation.
Consider a processes, which demands 50MB of contiguous memory from kernel. (keep in mind contiguous). Obviously, the kernel would have allocated
pages randomly to different processes, and freed some of them. However, when we demand contiguous memory, it will have to look for a chunk
which satifies the processes demand. If it is not able to get such a memory, it will have to do a swap-out of some old memory pages and then allocate
contiguous ones. Even in such cases SWAP out would happen. Starting Kernel ver 2.6 and above, such fragmentation problems have reduced considerably.
However, if the system is running for a long time, such problems could still come.
See this example (vmstat output)
2016-10-29 03:55:32 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
2016-10-29 03:55:32 r b swpd free buff cache si so bi bo in cs us sy id wa st
2016-10-30 03:56:04 19 23 2914752 4692144 3344908 12162628 1660 1 8803 12701 4336 37487 14 7 40 38 0
2016-10-30 03:56:34 3 20 2889296 4977580 3345316 12026752 2109 2 8445 14665 4656 36294 12 7 46 34 0
2016-10-30 03:57:04 1 11 3418868 4939716 3347804 11536356 586 4744 2547 9535 3086 24450 6 3 59 33 0 <<<-----
2016-10-30 03:57:34 3 19 3456252 5449884 3348400 11489728 3291 13371 6407 17957 2997 22556 6 4 66 24 0
2016-10-30 03:58:04 7 6 4194500 5663580 3349552 10857424 2407 12240 3824 14560 2295 18237 4 2 65 29 0
2016-10-30 03:58:34 2 16 4203036 5986864 3348908 10838492 4601 16639 7219 18808 2575 21563 6 4 60 31 0
2016-10-30 03:59:04 3 14 4205652 6059196 3348760 10821448 6624 1597 9431 4357 1750 20471 6 2 60 31 0
2016-10-30 03:59:34 2 24 4206968 6053160 3348876 10777216 5221 2067 10106 7377 1731 19161 3 3 62 32 0
2016-10-30 04:00:04 0 13 4205172 6005084 3348932 10785896 6236 1609 10330 6264 1739 20348 4 2 67 26 0
2016-10-30 04:00:34 4 11 4206420 5996396 3348976 10770220 6554 1253 10382 4896 1964 42981 10 5 58 27 0
2016-10-30 04:01:04 6 4 4177176 5878852 3348988 10825840 8682 765 10126 2716 1731 32949 8 4 69 19 0
@2016-10-30 03:57:04, we see that there is still good amount of free RAM available. However, even then swap out happened. We checked the process tree at this point, and we did not see any process coming up which would demand such high amount of memory (more than free memory). The obvious suspicion was Situation 2 described above.
We checked buddyinfo and zoneinfo logs above (Use echo m > /proc/sysrq-trigger to check these, output goes into syslogs).
For a normal system of ours, the comparison of zone info goes this this.
And graphs for cache/free/low mem is also mentioned below
Looking at the info, it is clear that there is memory fragmentation in node 0 & node 1 normal (Node it is NUMA based machine, hence multiple nodes (see numactl to check info for your system)).
Memory fragmentation is also a reason why swap usage can increases even when free memory is there.