4

When my service fails I get the the following output:

Nov 06 10:23:55 irgprod2 systemd[1]: emsxsub.service: Unit entered failed state.
Nov 06 10:23:55 irgprod2 systemd[1]: emsxsub.service: Failed with result 'exit-code'.
Nov 06 10:23:58 irgprod2 systemd[1]: emsxsub.service: Service hold-off time over, scheduling restart.
Nov 06 10:23:58 irgprod2 systemd[1]: Stopped EMSX API Subscription.

There is no explanation of what has gone wrong or why the service wasn't restarted, no mention of "start-limit" or anything. This is the process:

[Unit]
Description=EMSX API Subscription
After=multi-user.target

[Service]
Type=simple
Restart=always
RestartSec=3
User=root
Environment=PYTHONPATH=/k/svn_prod
Environment=LD_LIBRARY_PATH=/opt/lib
ExecStart=/usr/bin/python /k/svn_prod/com_wrappers/bbg_emsxsub.py
RemainAfterExit=no

[Install]
WantedBy=multi-user.target

I have tried without the User arg, without the RemainAfterExit arg, etc. The systemctl process is enabled, and the process runs and restarts successfully when I execute "service emsxsub restart". It is only when it fails that systemd schedules the restart and then stops the service without successfully restarting it. I cannot find anybody else with this problem - other issues seem to be a "start-limit" error which is not the case here. Has anybody seen this issue before? I cannot figure out why it all works when it is manually restarted but the service is unable to restart itself on failure.

Here is the journalctl output:

Nov 06 10:23:55 irgprod2 python[18451]: Traceback (most recent call last):
Nov 06 10:23:55 irgprod2 python[18451]:   File "/k/svn_prod/com_wrappers/bbg_emsxsub.py", line 533, in <module>
Nov 06 10:23:55 irgprod2 python[18451]:     main()
Nov 06 10:23:55 irgprod2 python[18451]:   File "/k/svn_prod/com_wrappers/bbg_emsxsub.py", line 505, in main
Nov 06 10:23:55 irgprod2 python[18451]:     raise RuntimeError()  # raise error so process restarts
Nov 06 10:23:55 irgprod2 python[18451]: RuntimeError
Nov 06 10:23:55 irgprod2 systemd[1]: emsxsub.service: Main process exited, code=exited, status=1/FAILURE
Nov 06 10:23:55 irgprod2 systemd[1]: emsxsub.service: Unit entered failed state.
Nov 06 10:23:55 irgprod2 systemd[1]: emsxsub.service: Failed with result 'exit-code'.
Nov 06 10:23:58 irgprod2 systemd[1]: emsxsub.service: Service hold-off time over, scheduling restart.
Nov 06 10:23:58 irgprod2 systemd[1]: Stopped EMSX API Subscription.

Output of systemctl status emsxsub.service:

* emsxsub.service - EMSX API Subscription
   Loaded: loaded (/lib/systemd/system/emsxsub.service; enabled; vendor preset: enabled)
   Active: inactive (dead) (Result: exit-code) since Wed 2018-11-07 08:18:49 EST; 40s ago
  Process: 3936 ExecStart=/usr/bin/python /k/svn_prod/com_wrappers/bbg_emsxsub.py (code=exited, status=1/FAILURE)
 Main PID: 3936 (code=exited, status=1/FAILURE)

Nov 07 08:18:45 irgprod2 systemd[1]: emsxsub.service: Unit entered failed state.
Nov 07 08:18:45 irgprod2 systemd[1]: emsxsub.service: Failed with result 'exit-code'.
Nov 07 08:18:49 irgprod2 systemd[1]: emsxsub.service: Service hold-off time over, scheduling restart.
Nov 07 08:18:49 irgprod2 systemd[1]: Stopped EMSX API Subscription.

Output of systemctl show emsxsub.service | grep -i restart:

[trading@irgprod2 ~]$ systemctl show emsxsub.service | grep -i restart
Restart=always
RestartUSec=3s

systemctl version:

systemd 229
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN

Distro details:

[trading@irgprod2 ~]$ cat /etc/*-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.3 LTS"
NAME="Ubuntu"
VERSION="16.04.3 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.3 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
  • 2
    This is odd... Can you try systemctl show emsxsub.service | grep -i restart to confirm the Restart configuration is correctly recognized in systemd's internal state for that unit? Also, please update with the distro you're using and systemd version, that might be useful in the investigation... – filbranden Nov 7 '18 at 14:19
  • 1
    Actually, I wanted to see systemctl show not systemctl status... See my comment above again. systemctl show will display internal state. – filbranden Nov 7 '18 at 14:54
  • 1
    Welcome ,you can add Environment=SYSTEMD_LOG_LEVEL=debug to get more informations. See Diagnosing a service – GAD3R Nov 7 '18 at 15:06
  • 2
    I can't reproduce your issue... Tried it on Ubuntu 16.04.5 with systemd 229-4ubuntu21.6 and I always see Active: activating (auto-restart). One more thing you could check is systemctl show emsxsub.service | grep StartLimit since those settings are related to restarts (but the default settings don't seem to cause this issue.) While looking at your unit definition today, I noticed you seem to have a circular loop, since you have After=multi-user.target (but I couldn't reproduce the issue with it anyways.) Maybe try removing that, or look for circular dependency warnings in the logs. – filbranden Nov 8 '18 at 8:28
  • 1
    Thanks Filipe, the circular dependency seems to have been the issue as removing "After=multi-user.target" has resolved the issue. – John Nov 9 '18 at 15:16
1

Removing After=multi-user.target solved the issue. Filipe's observation of a circular dependency seems to have been the correct call.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.