Debian wheezy AMD64, 3 cores, 2560mb ram (virtual machine). 1 virtual drive on ext4, running on KVM.
I decided to move it from raw to qcow2. Everything is ok and everything was going ok during migration. But it did not resolve my problem. It has been always slow on I/O and after moving to qcow2 it is still slow.
Here is a screenshot with graphs from munin it shows up:
http://wikisend.com/download/798312/bad_kvm_virtual.png
Services are lagging. On iotop --only almost nothing shows.
vmstat 1 shows this:
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 0 253232 95936 883488 0 0 9 23 145 82 5 4 87 4
1 0 0 253108 95936 883488 0 0 0 0 597 1045 4 3 94 0
0 0 0 253108 95936 883488 0 0 0 0 596 1228 4 4 93 0
0 0 0 252488 95936 883488 0 0 136 0 424 2216 4 4 90 2
0 0 0 244800 95936 885540 0 0 1924 0 629 1387 4 5 85 7
0 0 0 244800 95936 885548 0 0 0 0 408 2576 3 4 93 0
1 0 0 244800 95936 885548 0 0 0 0 388 1852 3 3 94 0
0 0 0 244800 95936 885548 0 0 0 0 387 2077 4 4 92 0
0 0 0 244908 95952 885540 0 0 0 788 623 2146 3 5 73 19
0 0 0 244948 95952 885544 0 0 0 0 477 1417 3 2 94 0
0 0 0 245024 95952 885544 0 0 0 12 527 2356 3 4 92 0
0 0 0 245024 95952 885544 0 0 0 0 406 1929 3 4 93 0
0 0 0 245024 95952 885544 0 0 0 572 319 2273 3 4 94 0
1 0 0 244900 95956 885540 0 0 0 80 398 2296 4 2 87 7
0 0 0 244900 95956 885544 0 0 0 0 334 2497 3 3 93 0
0 0 0 244900 95956 885544 0 0 0 0 315 2090 3 3 94 0
0 0 0 244900 95956 885544 0 0 0 0 330 2291 3 3 94 0
0 0 0 244900 95956 885544 0 0 0 0 385 2256 4 4 92 0
0 0 0 230516 95956 885544 0 0 60 0 570 1702 4 4 92 0
2 0 0 230448 95956 885604 0 0 0 0 341 2280 4 4 93 0
0 0 0 230448 95956 885604 0 0 0 0 327 2478 3 3 93 0
0 0 0 230448 95956 885604 0 0 0 0 335 2288 4 3 93 0
0 0 0 230200 95964 885596 0 0 0 180 344 2123 4 3 88 5
0 0 0 230200 95964 885604 0 0 0 0 450 1926 3 5 93 0
0 0 0 230200 95964 885604 0 0 0 0 390 2202 4 4 92 0
0 0 0 230200 95964 885604 0 0 0 0 325 2282 4 4 92 0
0 0 0 230200 95964 885604 0 0 0 0 332 2293 4 3 93 0
0 0 0 230200 95964 885604 0 0 0 0 423 1778 4 3 93 0
0 0 0 230200 95968 885604 0 0 0 4 375 2261 3 4 92 1
0 0 0 230200 95968 885604 0 0 0 0 319 2279 3 4 93 0
0 0 0 230200 95968 885604 0 0 0 0 397 2609 3 4 93 0
0 0 0 230200 95968 885604 0 0 0 0 328 2481 3 4 93 0
So and here is my problem. It shows - very high I/O Time, but nothing shows as a problem inside the virtual machine. No process is eating too much resources.
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 1 0 177880 96116 886304 0 0 0 0 451 1908 4 3 61 32
0 1 0 177880 96116 886304 0 0 0 0 400 1735 3 3 63 31
0 1 0 177880 96116 886304 0 0 0 0 359 2143 4 4 62 30
0 1 0 177284 96124 886272 0 0 0 216 813 1283 11 7 55 27
0 1 0 177316 96124 886300 0 0 0 0 439 2411 3 4 61 31
2 0 0 190344 96124 886316 0 0 0 76 1272 1986 14 7 77 3
0 0 0 190988 96124 886288 0 0 0 36 859 1714 10 6 84 0
2 0 0 191112 96124 886304 0 0 0 40 749 1494 12 4 84 0
0 0 0 191160 96124 886296 0 0 0 40 437 2156 12 3 84 0
1 0 0 191160 96124 886300 0 0 0 164 653 1466 8 3 80 9
0 0 0 191160 96124 886320 0 0 0 36 588 2428 10 4 85 0
1 0 0 191284 96124 886260 0 0 0 36 567 1534 10 3 86 0
0 0 0 191432 96124 886292 0 0 0 56 669 1436 16 3 81 0
1 0 0 191556 96124 886280 0 0 0 52 779 1212 14 4 82 0
0 0 0 191432 96128 886320 0 0 0 180 425 2167 9 4 69 18
0 0 0 191432 96128 886312 0 0 0 48 689 1236 11 4 84 0
1 0 0 190860 96128 886320 0 0 0 56 690 1713 22 4 74 0
1 0 0 190364 96128 886284 0 0 0 48 735 1398 20 5 75 0
1 0 0 190240 96128 886284 0 0 0 56 795 1713 24 3 73 0
0 0 0 189744 96136 886288 0 0 0 208 704 1057 9 4 73 15
0 0 0 189744 96136 886292 0 0 0 84 832 1405 20 5 75 0
1 0 0 189744 96136 886316 0 0 0 56 631 1178 12 4 84 0
0 2 0 189372 96136 886316 0 0 0 1588 478 2171 6 3 65 26
0 2 0 189372 96136 886316 0 0 0 0 353 2503 3 4 63 30
0 2 0 189372 96136 886316 0 0 0 0 441 2106 3 3 63 31
0 2 0 189248 96140 886312 0 0 0 792 367 2328 3 4 37 56
0 2 0 189248 96140 886312 0 0 0 0 360 2307 3 4 32 61
0 2 0 189248 96140 886312 0 0 0 0 354 2140 3 3 33 61
0 2 0 189248 96140 886312 0 0 0 0 354 2320 2 4 32 62
0 2 0 189248 96140 886312 0 0 0 0 473 1691 4 4 31 61
1 2 0 189248 96140 886312 0 0 0 468 339 2295 3 4 40 52
0 2 0 189248 96140 886312 0 0 0 0 312 2468 3 3 32 62
0 2 0 189248 96140 886312 0 0 0 0 552 1338 3 4 33 61
0 2 0 189248 96140 886312 0 0 0 0 615 1088 4 3 33 60
0 2 0 189248 96140 886312 0 0 0 0 560 1061 2 5 32 61
0 2 0 189248 96140 886312 0 0 0 348 316 2488 3 3 32 62
0 2 0 189248 96140 886312 0 0 0 0 497 1245 2 5 30 63
0 2 0 189248 96140 886312 0 0 0 0 374 2037 3 3 31 63
0 1 0 189248 96140 886312 0 0 0 440 364 2152 4 3 42 52
0 1 0 189248 96140 886312 0 0 0 0 423 1996 3 4 63 31
0 1 0 189248 96140 886312 0 0 0 0 375 2150 3 4 62 31
0 1 0 189248 96140 886312 0 0 0 0 596 1053 3 4 63 30
0 1 0 189248 96148 886312 0 0 0 1796 344 2300 5 4 61 30
0 1 0 189248 96148 886316 0 0 0 0 324 2486 3 4 61 32
0 1 0 189248 96148 886316 0 0 0 0 325 2294 4 3 62 32
0 1 0 209740 96152 886308 0 0 0 8 503 1446 4 4 63 29
0 0 0 209864 96152 886308 0 0 0 0 601 1082 3 4 82 11
0 0 0 209864 96152 886308 0 0 0 1772 473 1751 4 3 93 0
0 0 0 209864 96152 886312 0 0 0 0 532 1299 3 4 93 0
0 0 0 210644 96152 886312 0 0 0 12 586 2024 3 4 90 0
0 1 0 210660 96156 886312 0 0 0 152 497 2271 3 5 83 8
0 0 0 210536 96156 886316 0 0 0 4 455 2088 4 3 84 8
0 0 0 210536 96156 886316 0 0 0 1620 375 2518 4 5 92 0
Physical server is running SW RAID 6, all drives are OK. All drives are 7200 RPM WD RE4 from different date of birth.
Users has sometimes 3 second lags during using their services. Filesystem is only 57% of usage. Randomly, it is not in specific time.
Could be there some problem of not installed guest additions for kvm? (I have never seen some guest additions for linux or debian).
On the same server, it does not happen on other virtual machines, so I thing, there must be problem inside of this linux of this machine. Where do you think is a problem?
On one other virtual machine it shows this:
Could it be a problem of bad sectors on some harddrives? But checking dev/md1 did not show any problems.
What to do?
-----edit-----
It does use VirtIO, it does not use Sata emulation, look:
qcow2file? Test withfile drive.qcow2: output should beQEMU QCOW Image (v2), .... How is the disk cache mode? writeback, writetrough or none? I had I/O performance problems with kvm-qemu, but it was related to the backend of the physical host server; can you exclude that? Is the performance on the host server as expected?filebenchops/s went from ~111 to ~1920 on a test system I benchmarked...