generic methodology to debug ordering cycles in systemd

Question

I am aware of following thread and supposedly an answer to it. Except an answer is not an answer in generic sense. It tells what the problem was in one particular case, but not in general.

My question is: is there a way to debug ordering cycles in a generic way? E.g.: is there a command which will describe the cycle and what links one unit to another?

For example, I have following in journalctl -b (please disregard date, my system has no RTC to sync time with):

Jan 01 00:00:07 host0 systemd[1]: Found ordering cycle on sysinit.target/start
Jan 01 00:00:07 host0 systemd[1]: Found dependency on local-fs.target/start
Jan 01 00:00:07 host0 systemd[1]: Found dependency on cvol.service/start
Jan 01 00:00:07 host0 systemd[1]: Found dependency on basic.target/start
Jan 01 00:00:07 host0 systemd[1]: Found dependency on sockets.target/start
Jan 01 00:00:07 host0 systemd[1]: Found dependency on dbus.socket/start
Jan 01 00:00:07 host0 systemd[1]: Found dependency on sysinit.target/start
Jan 01 00:00:07 host0 systemd[1]: Breaking ordering cycle by deleting job local-fs.target/start
Jan 01 00:00:07 host0 systemd[1]: Job local-fs.target/start deleted to break ordering cycle starting with sysinit.target/start

where cvol.service (the one that got introduced, and which breaks the cycle) is:

[Unit]
Description=Mount Crypto Volume
After=boot.mount
Before=local-fs.target

[Service]
Type=oneshot
RemainAfterExit=no
ExecStart=/usr/bin/cryptsetup open /dev/*** cvol --key-file /boot/***

[Install]
WantedBy=home.mount
WantedBy=root.mount
WantedBy=usr-local.mount

According to journalctl, cvol.service wants basic.service, except that it doesn't, at least not obviously. Is there a command which would demonstrate where this link is derived from? And in general, is there a command, which would find the cycles and show where each link in the cycle originates?

Answer 1

You can visualise the cycle with the commands systemd-analyze verify , systemd-analyze dot and the GraphViz dot tool:

systemd-analyze verify default.target |&
perl -lne 'print $1 if m{Found.*?on\s+([^/]+)}' |
xargs --no-run-if-empty systemd-analyze dot |
dot -Tsvg >cycle.svg

You should see something like this:

Here you can see the cycle: c.service->b.service->a.service->c.service

Color legend: 
    black     = Requires
    dark blue = Requisite
    dark grey = Wants
    red       = Conflicts
    green     = After

Links:

• systemd-analyze(1)
• dot(1)

Answer 2

Is there a command which would demonstrate where this link is derived from?

The closest you can do is systemctl show -p Requires,Wants,Requisite,BindsTo,PartOf,Before,After cvol.service, which will show the resulting (effective) dependency lists for a given unit.

is there a command, which would find the cycles and show where each link in the cycle originates?

To my knowledge, there is no such command. Actually systemd offers nothing to aid in debugging ordering cycles (sigh).

According to journalctl, cvol.service wants basic.service, except that it doesn't, at least not obviously.

First, the requirement dependencies (Wants=, Requires=, BindsTo= etc.) are independent of ordering dependencies (Before= and After=). What you see here is an ordering dependency cycle, i. e. it has nothing to do with Wants= etc.

Second, there is a number of "default dependencies" created between units of certain types. They are controlled by DefaultDependencies= directive in the [Unit] section (which is enabled by default).

In particular, unless this directive is explicitly disabled, any .service-type unit gets implicit Requires=basic.target and After=basic.target dependencies, which is exactly what you see. This is documented in systemd.service(5).

Answer 3

step 1: run verification command for default.target

systemd-analyze verify default.target

step 2: observe which service or target mentioned in the message "systemd Breaking ordering cycle by deleting job" and display it's complete dependency list

systemctl show -p Requires,Wants,Requisite,BindsTo,PartOf,Before,After <service or target name mentioned in the "breaking cycle" message>

step 3: look at "after" and "before" groups inside the service or target file usually defined in

/lib/systemd/system

and find the services or targets well known to be sequential but are in outbound order for this one.

example:

dbus.service

is usually marked "after"

multi-user.target

but "before"

sockets.target

such dependency could be easily observed by calling

systemctl list-dependencies default.target

however if the file

/lib/systemd/system/dbus.service

contain lines like:

Before=multi-user.target

or

After=sockets.target

or both simultaneously, means dbus.service is defined outbound and it's causing an systemd endless cycle.

the cure is simple - change word "After" to "Before" and vice-versa if necessary.