On a mid-class CentOS 6.4/64 server with 32 GB RAM, 3 TB free disk space, operating as KVM hypervisor, I start copying a 200 GB file to a destination in the same local filesystem. In fact, this file is a KVM virtual disk image (corresponding to a shut down VM). Other 12 VMs are up & working normally on this same machine.
I start with plenty of headroom to go:
[root@myserver]$ free total used free shared buffers cached Mem: 32847956 16722708 16125248 0 63756 407740 -/+ buffers/cache: 16251212 16596744 Swap: 16383992 0 16383992
But as the copy progresses, memory usage starts to grow steadily until it hits swap. Of course, this slows down everything now... the copy finally ends after ~30 minutes. At the end, my memory looks like:
[root@myserver]$ free total used free shared buffers cached Mem: 32847956 32643564 204392 0 24392 23213400 -/+ buffers/cache: 9405772 23442184 Swap: 16383992 12057880 4326112
Looking what processes are now using swap, I observe it several of the qemu-kvm instances. So now the performance of the server is suffering, as many if not all the VMs are now swapping. I don't find a way to get swap back to zero (its otherwise normal condition) without having to bring this production server to a reboot.
What can cause this ? How can a simple cp process eat that much memory and how can this be avoided ? Any comments ?