Service file exists but is not found by systemd

Question

I have created a systemd service file and placed it in /etc/systemd/system/anfragen-3dkonfig-mapper.service. I ran systemctl daemon-reload, systemctl daemon-reexec and rebooted the system.

• systemctl enable anfragen-3dkonfig-mapper results in

  Failed to enable unit: Unit file anfragen-3dkonfig-mapper.service does not exist.
  

• systemctl start anfragen-3dkonfig-mapper results in

  Failed to start anfragen-3dkonfig-mapper.service: Unit anfragen-3dkonfig-mapper.service not found.
  

• ls -lh /etc/systemd/system/anfragen-3dkonfig-mapper.service outputs

  -rw-r--r--. 1 root root 440 Mar 19 12:08 /etc/systemd/system/anfragen-3dkonfig-mapper.service
  

• cd /root && systemd-analyze verify anfragen-3dkonfig-mapper.service has an exit code of 0 and prints no output.

• mount shows

  /dev/sda2 on / type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
  

• There are no other mounts touching /usr or /etc.

• The contents of the service file are:

  [Unit]
  Description=Anfragen 3D Konfigurations Mapper Service
  After=network.target
  
  [Service]
  Restart=always
  ExecStartPre=-/usr/bin/podman stop anfragen-3dkonfig-mapper
  ExecStartPre=-/usr/bin/podman rm anfragen-3dkonfig-mapper
  ExecStart=/usr/bin/podman run --rm --name anfragen-3dkonfig-mapper-app -p 10010:10000 anfragen-3dkonfig-mapper-app:0.0.1
  ExecStop=/usr/bin/podman stop anfragen-3dkonfig-mapper
  
  [Install]
  WantedBy=multi-user.target
  

• All above commands were run as the root user.

• Operating System: CentOS Linux release 8.0.1905 (Core)
• Systemd version: 239
• Linux kernel: Linux version 4.18.0-80.11.2.el8_0.x86_64 ([email protected]) (gcc version 8.2.1 20180905 (Red Hat 8.2.1-3) (GCC))
• I vaguely remember having a similar problem with another service file some months ago which just magically started working after a few hours of poking around and renaming the service file back and forth.

I'm interested in two things:

• How does one debug such a problem?
• What is wrong?

Answer 1

As hinted at by @JdeBP wrong SELinux file labels are the reason for the behavior. The . character in the output of ls indicates that there is a security context set for the file. So be attentive to the . in the ls output!

cd /etc/systemd/system && ls -lhZ some-other-service.service anfragen-3dkonfig-mapper.service prints

-rw-r--r--. 1 root root unconfined_u:object_r:admin_home_t:s0        440 Mar 19 12:08 anfragen-3dkonfig-mapper.service
-rw-r--r--. 1 root root unconfined_u:object_r:systemd_unit_file_t:s0 457 Feb 24 11:42 some-other-service.service

It can be seen that the other service file has the systemd_unit_file_t label, while the broken service doesn't. This can be fixed with restorecon anfragen-3dkonfig-mapper.service. After this the labels look as follows:

-rw-r--r--. 1 root root unconfined_u:object_r:systemd_unit_file_t:s0 440 Mar 19 12:08 anfragen-3dkonfig-mapper.service
-rw-r--r--. 1 root root unconfined_u:object_r:systemd_unit_file_t:s0 457 Feb 24 11:42 some-other-service.service

systemd now behaves as expected.

Answer 2

-rw-r--r--. 

SELinux restrictions are making life complex for you.

Answer 3

I just spent an hour chasing this issue after moving some of the service files around. The symlink in /lib/systemd/system was pointing to the correct file, but the symlink in /etc/systemd/system was not (its target no longer existed). I removed this offending (broken) symlink, replaced it with the correct one, and it worked.

Answer 4

Had similar issue, we use rootless podman on centos7.

Container were not started after reboot, the services were not found. But before reboot the services were enabled and present in /etc/systemd/system. Services were created via syslink ln -s from /home/user to /etc/systemd

When you run after reboot Systemctl status nameOfService.service it returns "Unit servicename.service could not be found."

When you run systemctl daemon-reload the service is present again.

One possible solution is to create a syslink ln -s from /root/ to /etc/systemd/system - after reboot your service will still exist.

Better solution

• Create your own service that executes daemon-reload and starts your service

Create startServiceOnBoot.sh script with daemon-reload and start of your service

    #!/bin/bash 
    sudo systemctl daemon-reload    
    sudo systemctl start nameOfService.service

Make sh script executable chmod +x startServiecOnBoot.sh

Create your service /etc/systemd/system/serviceStarter.service

[Unit]
Description=Daemon Reloader
...

[Service]
ExecStart=/home/user/startServiecOnBoot.sh
...

[Install]
WantedBy=multi-user.target

Enable service on boot systemctl enable serviceStarter.service you can also check if starting your new service serviceStarter.service actually starts nameOfService.service aswell.

I tried After= and .timer with services in systemd but were not successfull. For debugging $systemd-analyze blame