What I have:
________ enp7s0f0------------>|router| no IP addr -------- | | | Xen Dom0 | | | xenbr0 192.168.169.169/24 | | _________________________ |---vif1.0-------->|Old guest VM (ubuntu14)| (eth0 192.168.169.69) | ------------------------ | ________________________ |---vif6.0-------->|New Guest VM (debian10)| (eth0 192.168.169.42) ------------------------
The back story: I built a Xen Dom0 many years ago and installed one VM, lets call it "old guest", running Ubuntu 14. Life is good.
It is now old, so I need a new one, but Xen Dom0 is too old to update also, so I moved "oldguest" off, installed a new Xen Dom0 using Debian 10, and moved the "oldguest" back on. Now I need to build a new VM, call it "newguest" and migrate data. I do so using:
xen-create-image --hostname=newguest --memory=4G --vcpus=2 --partition=/etc/xen-tools/partitions.d/newguest --dist=buster --dir=/home/xen --ip=192.168.169.42 --broadcast=192.168.169.255 --gateway=192.168.169.1 --nodhcp --netmask=255.255.255.0 --nameserver=18.104.22.168
Seems to go ok, I then ssh into the new guest, create a user for myself, and do
apt-get update ; apt-get upgrade
Hrm, thats odd, it takes like, FOREVER AND A DAY for apt-get to come back.
Check networking: seems fine.
I decide to do a non-scientific test and grab a file, I think it could be the net? Ops, do not have wget:
apt-get install wget
THE ABOVE COMMAND TAKES LIKE 10 MINUTES. I ssh into the vm from another host. CPU load seems moderately high... lets try a different test: I have yet another host connected to the node called "router" in my diagram that is also running linux. I perform the wget:
65kB/s average download.
I go to "oldguest" and perform the same test:
11MB/s. OK, that is the correct speed to get.
Observations: If I reboot "newguest" and do NOT perform any action on "oldguest", I can get 11MB/s with the same wget test! Hrm, this makes me think it is networking on the Dom0. If I go to "oldguest" and perform the same test I get 11MB/s, BUT, if I go back to "newguest" and run the test again, it drops to around 50-60kB/s.
Note, I have even tried the download from "newguest" and then ssh into "oldguest" while it is running. The instant "oldguest" gets a packat, the speed on "newguest" falls to the floor.
One final note on speed tests:
If I copy/scp a file from "oldguest" to "newguest", I get 60MB/s. So this seems to be an issue with packets traversing Dom0 to "newguest", or a problem in my network configuration.
Steps taken to try to solve:
Rechecked all network configs. Check routing, all guests have the node marked "router" as the default gateway. Checked processes, it seems like I am not CPU bound anywhere.
Rechecked my config against this.
Have read this article on interfaces.
Have read this post, which is actually the opposite of my config.
auto enp7s0f0 iface enp7s0f0 inet manual auto xenbr0 iface xenbr0 inet static bridge_ports enp7s0f0 address 192.168.169.169 broadcast 192.168.169.255 netmask 255.255.255.0 gateway 192.168.169.1
Note, in /etc/xen/newguest we see:
vif = [ 'ip=192.168.169.42 ,mac=00:16:3E:27:99:CD,bridge=xenbr0' ]
in /etc/xen/oldguest we see:
vif = [ 'ip=192.168.169.69 ,mac=00:16:3E:B1:AC:98,bridge=xenbr0' ]
auto eth0 iface eth0 inet static address 192.168.169.42 gateway 192.168.169.1 netmask 255.255.255.0 broadcast 192.168.169.255
Old Guest /etc/network/interfaces
auto eth0 iface eth0 inet static address 192.168.169.69 gateway 192.168.169.1 netmask 255.255.255.0 broadcast 192.168.169.255
Update, after reading this page, I have been doing top and network and xentop testing. It does not seem to be a Dom0 bottleneck. See this screenshot during file transfer, there does not see to be any real problem, and xentop shows 1% CPU utilization also.
Any help in diagnosis would be great. I am thinking it is the hypervisor, but am not sure what to look at next.