4

On a mid-class CentOS 6.4/64 server with 32 GB RAM, 3 TB free disk space, operating as KVM hypervisor, I start copying a 200 GB file to a destination in the same local filesystem. In fact, this file is a KVM virtual disk image (corresponding to a shut down VM). Other 12 VMs are up & working normally on this same machine.

I start with plenty of headroom to go:

[root@myserver]$ free
             total       used       free     shared    buffers     cached
Mem:      32847956   16722708   16125248          0      63756     407740
-/+ buffers/cache:   16251212   16596744
Swap:     16383992          0   16383992

But as the copy progresses, memory usage starts to grow steadily until it hits swap. Of course, this slows down everything now... the copy finally ends after ~30 minutes. At the end, my memory looks like:

[root@myserver]$ free
             total       used       free     shared    buffers     cached
Mem:      32847956   32643564     204392          0      24392   23213400
-/+ buffers/cache:    9405772   23442184
Swap:     16383992   12057880    4326112

Looking what processes are now using swap, I observe it several of the qemu-kvm instances. So now the performance of the server is suffering, as many if not all the VMs are now swapping. I don't find a way to get swap back to zero (its otherwise normal condition) without having to bring this production server to a reboot.

What can cause this ? How can a simple cp process eat that much memory and how can this be avoided ? Any comments ?

Thanks

  • "this file is a KVM virtual disk image" - hmm, maybe logical volumes would behave better in a scenario like this? – Martin von Wittich Sep 24 '13 at 19:28
  • As Warren Young said, swap is not that bad. If it's not swapping back to memory, nothing is using it. If you really want to (and you have enough mem) you could try echo 1 > /proc/sys/vm/drop_caches; swapoff -a; swapon -a to get the counters back to where they were. – Matt Sep 24 '13 at 23:51
  • or nocache – Matt Sep 25 '13 at 0:07
5

Getting swap back to 0 is not a useful goal.

There is nothing ipso facto wrong about having things in swap. It is quite possible for a program to load resources it doesn't actually use, and for the kernel to notice this and swap them out, freeing the memory for use by programs that can actually make use of it right now. This situation comes up a lot in today's modern bloatware world, where programs depend so much on huge libraries others provide, even though they need only a tiny fraction of the library's full capability.

The only hard numbers you have provided — 200 GB in 30 minutes — also looks quite good to me. That's 114 MByte/sec, which is an impressive copy rate, considering that you're copying a file within a single physical volume. It wasn't that long ago that 100 MByte/sec on purely sequential reads was pretty impressive. You're managing better than that with interleaved reads and writes!

Bottom line, I think you're barking up the wrong tree.

0

Lower the value in /proc/sys/vm/swappiness to sacrifice the filesystem cache in favor of not swapping programs.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.