When running VMs on an ESXi host, you can overcommit the memory.
Example: host == 20GB physical RAM
The amount of RAM allocated to each VM is
- VM1 == 10gb
- VM2 == 10gb
- VM3 == 10gb
- VM4 == 10gb
The result is even though there is only 20GB of ram available, We have actually allocated 40GB of it.
There is no problem with this if each machine uses and average of 5GB of "active" memory (inactive memory is swapped out in a few different manners).
The issue is, VM1 does a big file operation, the kernel writes an additional 5GB to it's RAM. As this is just written, the esx host considers it as active memory, and starts swapping out memory from the other running VMs (thus hitting their performance).
Flushing the cache after a big operation (using is not a great option as the host has already started penalising the other VMs.)
The above is a simplification (the actual problem VM has 32GB of RAM assigned), but essentially:
How do I limit the max size of the disk cache on a linux kernel.