How to troubleshoot systemd's "contradicts existing jobs" error message

Question

Looking through my system logs, I see the following message in /var/log/messages:

Sep  2 15:29:15 <myhostname> systemd[1]: Requested transaction contradicts existing jobs: File exists

So, my question is: How do I troubleshoot this error? What steps should I take to understand what this means, what is causing it, whether it is something to be worried about, and if so, how to correct it?

And, what is the meaning of the message "Requested transaction contradicts existing jobs"? I suspect there must be conceptual background that I am missing. In any case, does this error message suggest some candidate next steps to diagnose the cause of this error message and understand what is causing it?

I've read the systemd docs, but I did not find them helpful with this. It'd be great to have a general guide on what to look for: I suspect this might be helpful to others who also experience the "... contradicts existing jobs" message.

Answer 1

To get more information on this error for troubleshooting purposes start by running the command:

$ journalctl -ab

This will dump some detailed logs out of the systemd journal.

Additionally, there was probably another error directly in front of that one telling you what service, specifically, is having an issue. Try diagnosing that service as well (using firewalld as an example):

$ systemctl status firewalld.service

Specifically that line comes from the systemd code for transaction activation:

http://cgit.freedesktop.org/systemd/systemd/commit/?id=75778e21dfeee51036d24501e39ea7398fabe502

And specifically, as to systemd transactions (from man systemd):

   systemd has a minimal transaction system: if a unit is requested to start up
   or shut down it will add it and all its dependencies to a temporary
   transaction. Then, it will verify if the transaction is consistent (i.e.
   whether the ordering of all units is cycle-free). If it is not, systemd will
   try to fix it up, and removes non-essential jobs from the transaction that
   might remove the loop. Also, systemd tries to suppress non-essential jobs in
   the transaction that would stop a running service. Finally it is checked
   whether the jobs of the transaction contradict jobs that have already been
   queued, and optionally the transaction is aborted then. If all worked out and
   the transaction is consistent and minimized in its impact it is merged with
   all already outstanding jobs and added to the run queue. Effectively this
   means that before executing a requested operation, systemd will verify that
   it makes sense, fixing it if possible, and only failing if it really cannot
   work.

So, combining all of this together, I'm guessing there is something else very near that log entry which is trying to start up but already has a lock file in place (potentially stale) which needs to be cleaned up.