delayed bash script execution after boot

Question

My Fedora Linux (presently 30) system hosts several virtual machines (VMs) configured to start up when host system boots. These are isolated servers supposed to be visible from the internet containing no important or critical data, so it does not matter if they are hacked (they'll then be wiped out and reloaded).

To make them visible, I must set the firewall in passthrough mode for the destination IPs on the virtual bridge. The commands are grouped in a bash script.

I noticed for the commands to be successful and effective that the VMs must be up and running, which means I must wait for the VM boots to finish, i.e. probably the time when they reach multi-user.target.

My problem is VM state can be tested in guest systems and my bash script must be run in the host system.

I already attempted to delay the bash script a fixed number of seconds but this does not seem to be always correct. VMs need a variable time to boot up (e.g. when they run their fsck on n-th boot) and I can't predict reliably the time when all are ready and running.

My goal is to operate my servers 24/7, automatically connecting them to internet in case of automatic restart of the host (after power failure or other events).

Presently, I launch this script manually (and somteimes I forget to do it).

Question: is there a way (systemd, cron, at, … ?) to launch my script after the VMs have settled in multi-user mode?

EDIT

According to @Michael D.'s comment, my description is not clear or accurate.

firewall-cmd always terminates "successfully" unless there are syntax errors in the command. Even configuration for non-existing devices is accepted without rejection by firewall. Thus I can't rely on bash $? to detect whether the command will have some effect or not. Monitoring return code will not help here.

Answer 1

After trials and errors, this is the solution I implemented.

I attached my script to a systemd service:

[Unit]
Description=Make VMs reachable from the internet
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
ExecStart=/home/…/virbr0-enable.sh

[Install]
Alias=makeVMvisible.service
WantedBy=multi-user.target

I made it wait until network is operational (after network-online.target) which, I assume, depends itself on firewalld.service, and also after multi-user.target has been reached. The latter may be superfluous (because I don't know the exact dependency relationships between targets and services) but it doesn't harm.

As such, it still didn't work. The systemd service was not faulty because a manual systemctl start makeVMvisible.service would put the virtual bridge in passthrough mode. A delay seems to be necessary: I added a sleep command at the beginning of the script.

From experiments, the threshold value for the delay is between 1 and 2 seconds. I decided for sleep 5 which does not make a big difference in elapsed time and is rather conservative.

However, I don't find my solution very satisfactory. Does any know of a better dependency than network-online.target?

Answer 2

Can you reach the VM from the host before the external IP address is passed through to it? If so, then you can use the remote functionality of systemctl to check the status of a VM's boot.

This requires that the VM be accessible using an ssh key from the host.

On the host you would run:

systemctl -H <vm name or ip> --quiet is-active multi-user.target

This exits successfully if the named unit is active.

So in bash you can do something like:

until systemctl -H <vm name or ip> --quiet is-active multi-user.target; do
    sleep 5
done

Now this will not proceed until the named unit on the given host is up.