3

Debian wheezy AMD64, 3 cores, 2560mb ram (virtual machine). 1 virtual drive on ext4, running on KVM.

I decided to move it from raw to qcow2. Everything is ok and everything was going ok during migration. But it did not resolve my problem. It has been always slow on I/O and after moving to qcow2 it is still slow.

enter image description here Here is a screenshot with graphs from munin it shows up: http://wikisend.com/download/798312/bad_kvm_virtual.png

Services are lagging. On iotop --only almost nothing shows.

vmstat 1 shows this:

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 0  0      0 253232  95936 883488    0    0     9    23  145   82  5  4 87  4
 1  0      0 253108  95936 883488    0    0     0     0  597 1045  4  3 94  0
 0  0      0 253108  95936 883488    0    0     0     0  596 1228  4  4 93  0
 0  0      0 252488  95936 883488    0    0   136     0  424 2216  4  4 90  2
 0  0      0 244800  95936 885540    0    0  1924     0  629 1387  4  5 85  7
 0  0      0 244800  95936 885548    0    0     0     0  408 2576  3  4 93  0
 1  0      0 244800  95936 885548    0    0     0     0  388 1852  3  3 94  0
 0  0      0 244800  95936 885548    0    0     0     0  387 2077  4  4 92  0
 0  0      0 244908  95952 885540    0    0     0   788  623 2146  3  5 73 19
 0  0      0 244948  95952 885544    0    0     0     0  477 1417  3  2 94  0
 0  0      0 245024  95952 885544    0    0     0    12  527 2356  3  4 92  0
 0  0      0 245024  95952 885544    0    0     0     0  406 1929  3  4 93  0
 0  0      0 245024  95952 885544    0    0     0   572  319 2273  3  4 94  0
 1  0      0 244900  95956 885540    0    0     0    80  398 2296  4  2 87  7
 0  0      0 244900  95956 885544    0    0     0     0  334 2497  3  3 93  0
 0  0      0 244900  95956 885544    0    0     0     0  315 2090  3  3 94  0
 0  0      0 244900  95956 885544    0    0     0     0  330 2291  3  3 94  0
 0  0      0 244900  95956 885544    0    0     0     0  385 2256  4  4 92  0
 0  0      0 230516  95956 885544    0    0    60     0  570 1702  4  4 92  0
 2  0      0 230448  95956 885604    0    0     0     0  341 2280  4  4 93  0
 0  0      0 230448  95956 885604    0    0     0     0  327 2478  3  3 93  0
 0  0      0 230448  95956 885604    0    0     0     0  335 2288  4  3 93  0
 0  0      0 230200  95964 885596    0    0     0   180  344 2123  4  3 88  5
 0  0      0 230200  95964 885604    0    0     0     0  450 1926  3  5 93  0
 0  0      0 230200  95964 885604    0    0     0     0  390 2202  4  4 92  0
 0  0      0 230200  95964 885604    0    0     0     0  325 2282  4  4 92  0
 0  0      0 230200  95964 885604    0    0     0     0  332 2293  4  3 93  0
 0  0      0 230200  95964 885604    0    0     0     0  423 1778  4  3 93  0
 0  0      0 230200  95968 885604    0    0     0     4  375 2261  3  4 92  1
 0  0      0 230200  95968 885604    0    0     0     0  319 2279  3  4 93  0
 0  0      0 230200  95968 885604    0    0     0     0  397 2609  3  4 93  0
 0  0      0 230200  95968 885604    0    0     0     0  328 2481  3  4 93  0

So and here is my problem. It shows - very high I/O Time, but nothing shows as a problem inside the virtual machine. No process is eating too much resources.

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 1  1      0 177880  96116 886304    0    0     0     0  451 1908  4  3 61 32
 0  1      0 177880  96116 886304    0    0     0     0  400 1735  3  3 63 31
 0  1      0 177880  96116 886304    0    0     0     0  359 2143  4  4 62 30
 0  1      0 177284  96124 886272    0    0     0   216  813 1283 11  7 55 27
 0  1      0 177316  96124 886300    0    0     0     0  439 2411  3  4 61 31
 2  0      0 190344  96124 886316    0    0     0    76 1272 1986 14  7 77  3
 0  0      0 190988  96124 886288    0    0     0    36  859 1714 10  6 84  0
 2  0      0 191112  96124 886304    0    0     0    40  749 1494 12  4 84  0
 0  0      0 191160  96124 886296    0    0     0    40  437 2156 12  3 84  0
 1  0      0 191160  96124 886300    0    0     0   164  653 1466  8  3 80  9
 0  0      0 191160  96124 886320    0    0     0    36  588 2428 10  4 85  0
 1  0      0 191284  96124 886260    0    0     0    36  567 1534 10  3 86  0
 0  0      0 191432  96124 886292    0    0     0    56  669 1436 16  3 81  0
 1  0      0 191556  96124 886280    0    0     0    52  779 1212 14  4 82  0
 0  0      0 191432  96128 886320    0    0     0   180  425 2167  9  4 69 18
 0  0      0 191432  96128 886312    0    0     0    48  689 1236 11  4 84  0
 1  0      0 190860  96128 886320    0    0     0    56  690 1713 22  4 74  0
 1  0      0 190364  96128 886284    0    0     0    48  735 1398 20  5 75  0
 1  0      0 190240  96128 886284    0    0     0    56  795 1713 24  3 73  0
 0  0      0 189744  96136 886288    0    0     0   208  704 1057  9  4 73 15
 0  0      0 189744  96136 886292    0    0     0    84  832 1405 20  5 75  0
 1  0      0 189744  96136 886316    0    0     0    56  631 1178 12  4 84  0
 0  2      0 189372  96136 886316    0    0     0  1588  478 2171  6  3 65 26
 0  2      0 189372  96136 886316    0    0     0     0  353 2503  3  4 63 30
 0  2      0 189372  96136 886316    0    0     0     0  441 2106  3  3 63 31
 0  2      0 189248  96140 886312    0    0     0   792  367 2328  3  4 37 56
 0  2      0 189248  96140 886312    0    0     0     0  360 2307  3  4 32 61
 0  2      0 189248  96140 886312    0    0     0     0  354 2140  3  3 33 61
 0  2      0 189248  96140 886312    0    0     0     0  354 2320  2  4 32 62
 0  2      0 189248  96140 886312    0    0     0     0  473 1691  4  4 31 61
 1  2      0 189248  96140 886312    0    0     0   468  339 2295  3  4 40 52
 0  2      0 189248  96140 886312    0    0     0     0  312 2468  3  3 32 62
 0  2      0 189248  96140 886312    0    0     0     0  552 1338  3  4 33 61
 0  2      0 189248  96140 886312    0    0     0     0  615 1088  4  3 33 60
 0  2      0 189248  96140 886312    0    0     0     0  560 1061  2  5 32 61
 0  2      0 189248  96140 886312    0    0     0   348  316 2488  3  3 32 62
 0  2      0 189248  96140 886312    0    0     0     0  497 1245  2  5 30 63
 0  2      0 189248  96140 886312    0    0     0     0  374 2037  3  3 31 63
 0  1      0 189248  96140 886312    0    0     0   440  364 2152  4  3 42 52
 0  1      0 189248  96140 886312    0    0     0     0  423 1996  3  4 63 31
 0  1      0 189248  96140 886312    0    0     0     0  375 2150  3  4 62 31
 0  1      0 189248  96140 886312    0    0     0     0  596 1053  3  4 63 30
 0  1      0 189248  96148 886312    0    0     0  1796  344 2300  5  4 61 30
 0  1      0 189248  96148 886316    0    0     0     0  324 2486  3  4 61 32
 0  1      0 189248  96148 886316    0    0     0     0  325 2294  4  3 62 32
 0  1      0 209740  96152 886308    0    0     0     8  503 1446  4  4 63 29
 0  0      0 209864  96152 886308    0    0     0     0  601 1082  3  4 82 11
 0  0      0 209864  96152 886308    0    0     0  1772  473 1751  4  3 93  0
 0  0      0 209864  96152 886312    0    0     0     0  532 1299  3  4 93  0
 0  0      0 210644  96152 886312    0    0     0    12  586 2024  3  4 90  0
 0  1      0 210660  96156 886312    0    0     0   152  497 2271  3  5 83  8
 0  0      0 210536  96156 886316    0    0     0     4  455 2088  4  3 84  8
 0  0      0 210536  96156 886316    0    0     0  1620  375 2518  4  5 92  0

Physical server is running SW RAID 6, all drives are OK. All drives are 7200 RPM WD RE4 from different date of birth.

Users has sometimes 3 second lags during using their services. Filesystem is only 57% of usage. Randomly, it is not in specific time.

Could be there some problem of not installed guest additions for kvm? (I have never seen some guest additions for linux or debian).

On the same server, it does not happen on other virtual machines, so I thing, there must be problem inside of this linux of this machine. Where do you think is a problem?

On one other virtual machine it shows this: enter image description here

Could it be a problem of bad sectors on some harddrives? But checking dev/md1 did not show any problems.

What to do?

-----edit-----

It does use VirtIO, it does not use Sata emulation, look: enter image description here

  • Can you add some more information, please? How big is the virtual drive? Are the virtio drivers installed and loaded: linux-kvm.org/page/Virtio, specially the Virtio block driver? How is the root filesystem mounted (I meand the mount options)? Is it a qcow2 file? Test with file drive.qcow2: output should be QEMU QCOW Image (v2), .... How is the disk cache mode? writeback, writetrough or none? I had I/O performance problems with kvm-qemu, but it was related to the backend of the physical host server; can you exclude that? Is the performance on the host server as expected? – chaos Sep 30 '15 at 9:35
  • Using LVM-backed virts instead of file-based virts might also be something to try out, filebench ops/s went from ~111 to ~1920 on a test system I benchmarked... – thrig Sep 30 '15 at 19:06
  • How did you "check md1"? – Wayne Walker Oct 1 '15 at 4:07
  • HDD size is 24GB, 11G free, used is 54% of filesystem. It works on ext4. No LVM at the virtual machine. Disk cache mode is same as on different virtual machines. Virtio, format qcow2, mode cache : default, io mode: default. Running on debian wheezy, host machine also debian wheezy. – MIrra Oct 1 '15 at 9:51
  • I was checking md1 with cat /proc/mdstat and with some command like echo check > /sys/block/md1/md/sync_action, it does once every month and no disk was removed from array. – MIrra Oct 1 '15 at 9:59
1

qcow2 is substantially slower than raw images. Using raw logical volumes is the most efficient use of hardware. Additionally raid10 would provide higher IOPS.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.