I've been experimenting with ZFS + DRBD + live migration (I want to understand it well enough to write my own automation scripts before I start playing with
ganeti again, and then openstack
cinder). I have the ZFS + DRBD (in dual-primary mode) working well for shared storage.
However, live migration is only partially working.
I have two hosts, with identical libvirt and drbd configurations, and even identical dedicated "volumes" pool for VM ZVOLs (both 2x1TB mirror pools - re-using some old disks from my old backup pool), and identical configurations for the VM (named "dtest")
"indra" is an AMD FX-8150 with 16GB RAM on an ASUS Sabertooth 990FX m/b
- cpu flags:
fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc extd_apicid aperfmperf eagerfpu pni pclmulqdq monitor ssse3 cx16 sse4_1 sse4_2 popcnt aes xsave avx lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4 nodeid_msr topoext perfctr_core perfctr_nb cpb hw_pstate vmmcall arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold
- cpu flags:
"surya" is an AMD Phenom II X4 940 with 8GB RAM on an ASUS M3A79-T DELUXE m/b
- cpu flags
fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl nonstop_tsc extd_apicid eagerfpu pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt hw_pstate vmmcall npt lbrv svm_lock nrip_save
- cpu flags
Both are running debian sid, with exactly the same versions of packages (incl.
libvirt* 2.0.0-1:amd64 and
qemu-system-x86 1:2.6+dfsg-3), and with the same liquorix kernel:
Linux indra 4.6-2.dmz.2-liquorix-amd64 #1 ZEN SMP PREEMPT Debian 4.6-3 (2016-06-19) x86_64 GNU/Linux Linux surya 4.6-2.dmz.2-liquorix-amd64 #1 ZEN SMP PREEMPT Debian 4.6-3 (2016-06-19) x86_64 GNU/Linux
The VM itself is running debian sid, with a stock debian 4.6.0-1 kernel:
Linux dtest 4.6.0-1-amd64 #1 SMP Debian 4.6.3-1 (2016-07-04) x86_64 GNU/Linux
I can start the VM on either host, and it works perfectly.
I can migrate a VM from surya to indra with no problems whatsoever. When I try to migrate the VM from indra to surya, the migration appears to complete successfully, but the VM hangs with 100% CPU usage (for the single core allocated to it).
It makes no difference whether the VM was started on indra and then migrated to surya (where it hangs) or if it was started on surya, migrated to indra (OK so far) and then migrated back to surya (hangs).
The only thing I can do with the VM when it hangs is
virsh destroy (force-shutdown) or
virsh reset (force-reboot).
I've tried disabling kvm_steal_time with:
<qemu:commandline> <qemu:arg value='-cpu'/> <qemu:arg value='qemu64,-kvm_steal_time'/> </qemu:commandline>
but that doesn't solve the problem.
Nothing gets logged on or from the VM itself. The only indication I get of any problem is the following message in
/var/log/libvirt/qemu/dtest.log on surya.
2016-07-18T12:56:55.766929Z qemu-system-x86_64: warning: TSC frequency mismatch between VM and host, and TSC scaling unavailable
This would be due to the
tsc_scale cpu feature - present on the 8150 CPU (indra), missing on the x4 940 (surya).
Anyone know what the problem is? Or how to fix it? or suggestions for debugging?
Is it even fixable, or is it a CPU bug in the several-generations-old Phenom II x4 940?