1

On my Xen host (systemd in Debian Buster, Xen 4.11), I usually start all guest VMs automatically. I added all my VM configs as link in /etc/xen/auto/ to the underlying config like

/etc/xen/auto/proxy.cfg -> /etc/xen/proxy.cfg
/etc/xen/auto/vm04.matrix.cfg -> /etc/xen/vm04.matrix.cfg
...

and sometimes some guest VMs start automatically on reboot of the DOM0 host and sometimes some fail and throw these errors in syslog:

$ grep "Starting Xen domain" /var/log/syslog
May 12 08:38:36 dom0 xendomains[1103]: Starting Xen domain proxy (from /etc/xen/auto/proxy.cfg)...done.
May 12 08:38:36 dom0 xendomains[1103]: Starting Xen domain vm04.matrix.de (from /etc/xen/auto/vm04.matrix.cfg)...failed.

If they fail, the errors in \var\log\syslog around the fail during boot are:

Starting Xen domain vm04.matrix (from /etc/xen/auto/vm04.matrix.cfg)...failed.
libxl: error: libxl_device.c:417:libxl__device_disk_set_backend: Disk vdev=xvda2 failed to stat: /dev/vg0/vm04.matrix-disk: No such file or directory
libxl: error: libxl_create.c:983:initiate_domain_create: Domain 4:Unable to set disk defaults for disk 0
libxl: error: libxl_domain.c:1034:libxl__destroy_domid: Domain 4:Non-existant domain
libxl: error: libxl_domain.c:993:domain_destroy_callback: Domain 4:Unable to destroy guest
libxl: error: libxl_domain.c:920:domain_destroy_cb: Domain 4:Destruction of domain failed

---

Starting Xen domain proxy (from /etc/xen/auto/proxy.cfg)...failed.
libxl: error: libxl_device.c:417:libxl__device_disk_set_backend: Disk vdev=xvda2 failed to stat: /dev/vg0/proxy-disk: No such file or directory
libxl: error: libxl_create.c:983:initiate_domain_create: Domain 1:Unable to set disk defaults for disk 0
libxl: error: libxl_domain.c:1034:libxl__destroy_domid: Domain 1:Non-existant domain
libxl: error: libxl_domain.c:993:domain_destroy_callback: Domain 1:Unable to destroy guest
libxl: error: libxl_domain.c:920:domain_destroy_cb: Domain 1:Destruction of domain failed

The files /etc/xen/proxy.cfg and /etc/xen/vm04.matrix.cfg are identical apart from the mount-points.

If I start xen create /etc/xen/vm04.matrix.cfg later manually it starts just fine, only not automatically during the boot of the host.

debugging

To debug the problem, I added this in /etc/default/grub

GRUB_CMDLINE_XEN_DEFAULT="dom0_mem=4096M,max:4096M loglvl=all guest_loglvl=all"

and called update-grub, but after a reboot the error messages in syslog are the same. I cannot find anything suspicious in dmesg either. No mentions of any of the guest hostnames. Only this shows, the problem:

# dmesg|grep dev|tail
[   11.348931] xen:xen_evtchn: Event-channel device installed
[   22.006020] device vif3.0 entered promiscuous mode
[   25.362878] device vif5.0 entered promiscuous mode
[   28.636912] device vif6.0 entered promiscuous mode
[ 1004.501654] device vif7.0 entered promiscuous mode
...

You only see, that the first three VMs Network-interfaces did start during boot this time and 1000s later I started the next VM manually on the console.

Xen loads just fine:

$ systemctl --type=service | grep -i xen
xen.service                        loaded active running LSB: Xen daemons                                                             
xendomains.service                 loaded active running LSB: Start/stop secondary xen domains

I think the best hint to the problem is that the volume is not there during boot:

/dev/vg0/proxy-disk: No such file or directory

maybe xen starts too early in the boot process?

systemctl cat xen.service|uniq
# /run/systemd/generator.late/xen.service
# Automatically generated by systemd-sysv-generator

[Unit]
Documentation=man:systemd-sysv-generator(8)
SourcePath=/etc/init.d/xen
Description=LSB: Xen daemons
Before=multi-user.target
Before=graphical.target
After=remote-fs.target

[Service]
Type=forking
Restart=no
TimeoutSec=5min
IgnoreSIGPIPE=no
KillMode=process
GuessMainPID=no
RemainAfterExit=yes
SuccessExitStatus=5 6
ExecStart=/etc/init.d/xen start
ExecStop=/etc/init.d/xen stop
systemctl cat xendomains.service|uniq
# /run/systemd/generator.late/xendomains.service
# Automatically generated by systemd-sysv-generator

[Unit]
Documentation=man:systemd-sysv-generator(8)
SourcePath=/etc/init.d/xendomains
Description=LSB: Start/stop secondary xen domains
Before=multi-user.target
Before=graphical.target
Before=corosync.service
Before=heartbeat.service
Before=libvirtd.service
After=network-online.target
After=remote-fs.target
After=xen.service
After=drbd.service
After=iscsi.service
After=openvswitch-switch.service
After=nfs-kernel-server.service
Wants=network-online.target

[Service]
Type=forking
Restart=no
TimeoutSec=5min
IgnoreSIGPIPE=no
KillMode=process
GuessMainPID=no
RemainAfterExit=yes
SuccessExitStatus=5 6
ExecStart=/etc/init.d/xendomains start
ExecStop=/etc/init.d/xendomains stop
ExecReload=/etc/init.d/xendomains reload

How can I further debug this?

5
  • could you please check with xl -vvvv to have more logs?
    – binarysta
    May 25, 2020 at 10:58
  • How can I call that during boot?
    – rubo77
    May 25, 2020 at 11:42
  • I added the service details, what is the TimeoutSec=5min referring to? maybe that is a problem, because a reboot of the mashine takes about 10minutes due to unknown reasons (i cannot see the console)
    – rubo77
    May 26, 2020 at 6:03
  • well, we can debug with journalctl -xb to see what happens during boot and also please check journalctl -u xen.service to investigate errors related to this service. TimeoutSec=: amount of time systemd will wait when stopping or starting the service before marking it as failed or forcefully killing it. We can make it infinity or restart on failure. lets first check if we can have more debug logs or not.
    – binarysta
    May 26, 2020 at 6:35
  • those journalctl commands don't show anything new, the same as in syslog already
    – rubo77
    May 26, 2020 at 7:49

3 Answers 3

2
+50

It looks like xendomains.service might be starting too early, before all the logical volumes in vg0 are activated. You might have to add/change some dependencies to as appropriate to your system configuration.

The current configuration is causing xendomains.service and thus the VMs to attempt starting before cluster components like corosync.service and heartbeat.service and VM management service libvirtd.service. If your vg0 is a clustered volume group that requires these cluster component services before it can activate, you should change xendomains.service to start After them, not Before.

And since libvirtd is the server-side daemon that actually handles starting and stopping other VMs, the attempt to start other VMs should probably happen only after libvirtd.service is started.

systemctl edit xendomains.service should open an empty temporary file in your designated editor, which will then automatically become an override file for the xendomains.service unit: /etc/systemd/system/xendomains.service.d/override.conf or something similar.

But the man page systemd.unit(5) says:

Dependencies (After=, etc.) cannot be reset to an empty list, so dependencies can only be added in drop-ins. If you want to remove dependencies, you have to override the entire unit.

So creating an override file won't work here. Instead, you would need to copy the existing auto-generated unit file to /etc/systemd/system/xendomains.service:

systemctl cat xendomains.service | uniq >/etc/systemd/system/xendomains.service

Then you can edit /etc/systemd/system/xendomains.service using your favorite editor, and finally run systemctl daemon-reload to make the changes effective. You should edit the list of dependencies to look like this:

[...the lines before this should be left as is...]
Description=LSB: Start/stop secondary xen domains
Before=multi-user.target
Before=graphical.target
After=corosync.service
After=heartbeat.service
After=libvirtd.service
After=network-online.target
After=remote-fs.target
After=xen.service
After=drbd.service
After=iscsi.service
After=openvswitch-switch.service
After=nfs-kernel-server.service
Wants=network-online.target
[...the lines after this should be left as is...]
3
  • @rubo77 these systemd scripts have been generated automatically by systemd-sysv-generator as you can see there are 3 same lines of Before=multi-user.target. Also please provide systemctl cat xendomains.service maybe we need to tune the dependencies.
    – binarysta
    May 26, 2020 at 13:54
  • @rubo77 Please see my edit. If systemctl edit xendomains.service opens an editor with an empty file, it's actually doing what it's generally supposed to do. But in this case, you'll need to override the entire autogenerated unit file so that you can remove the unwanted Before= dependencies.
    – telcoM
    Jun 2, 2020 at 5:09
  • I tried the override, but after a reboot now it sais: # systemctl status xendomains.service ● xendomains.service - LSB: Start/stop secondary xen domains Loaded: loaded (/etc/init.d/xendomains; static; vendor preset: enabled) Active: inactive (dead) it seems like it never started
    – rubo77
    Jun 3, 2020 at 4:22
0

I am not sure this is available in 4.11 I know in 6 it is:

xe pool-param-set uuid=UUID other-config:auto_poweron=true

When you do a "xe vm-list" that will give you the UUIDs.

2
  • I get xe: command not found
    – rubo77
    Jun 1, 2020 at 10:03
  • it is called xen in my system: xen vm-list | UUID ID name | d79abc4c-2ab0-4679-9435-b0001dca406b 24 proxy ... but there is no xen pool-param-set in xen 4.11
    – rubo77
    Jun 1, 2020 at 10:08
0

Workaround

If there is no solution to put the starting of all VMs, you can create a new script, that checks all configs in /etc/xen/auto and starts them manually if not started.

1. create a file /usr/local/sbin/xen-start-all-auto

#!/bin/bash
cd /etc/xen/auto/

for x in *; do
 d=$(echo $x|sed 's/.cfg$//g')
 echo check $d ...
 lvs|grep $d-disk|grep -v snap|grep -q -- '-ao'
 if [ $? == 1 ]; then
   xen create $x
 else
   echo ok
 fi
done

and make it executable:

sudo chmod +x /usr/local/sbin/xen-start-all-auto

2. create a systemd service that starts once

create a file /etc/systemd/system/xen-autostart.service

[Unit]
Description=starts all XEN vms in /etc/xen/auto if they are not started already 1 minute after system start

[Service]
TimeoutStartSec=infinity
ExecStartPre=/bin/sleep 60
ExecStart=/bin/bash /usr/local/sbin/xen-start-all-auto

[Install]
WantedBy=default.target

Update systemd with:

systemctl daemon-reload
systemctl enable xen-autostart
systemctl start xen-autostart &
2
  • Maybe interested of the bash script, this command would be enough: systemctl restart xendomains
    – rubo77
    Jun 3, 2020 at 4:25
  • no, systemctl restart xendomains also just starts the same "some" VMs
    – rubo77
    Aug 20, 2023 at 15:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .