34

Before all the unit files were in /etc/systemd/system/ but now some are showing up in /usr/lib/systemd/system (<- on CentOS, or /lib/systemd/system <- on Debian/Ubuntu), what is the difference between these folders?

29

This question is already answered in man 7 file-hierarchy which comes with systemd (there is also online version):

        /etc
           System-specific configuration.
 (…)
 VENDOR-SUPPLIED OPERATING SYSTEM RESOURCES
       /usr
            Vendor-supplied operating system resources. 
            Usually read-only, but this is not required. Possibly 
            shared between multiple hosts. This directory should not
            be modified by the administrator, except when installing 
            or removing vendor-supplied packages.

Basically, files that ships in packages downloaded from distribution repository go into /usr/lib/systemd/. Modifications done by system administrator (user) go into /etc/systemd/system/.

System-specific units override units supplied by vendors. Using drop-ins, you can override only specific parts of unit files, leaving the rest to vendor (drop-ins are available since the very beginning of systemd, but were properly documented only in v219; see man systemd.unit).

14

Background

If you look at the man page man systemd.unit it has a table that explains the differences. This is from a CentOS 7.x system.

   UNIT LOAD PATH
          Unit files are loaded from a set of paths determined during 
          compilation, described in the two tables below. Unit files found 
          in directories listed earlier override files with the same name 
          in directories lower in the list.

           Table 1.  Load path when running in system mode (--system).
           ┌────────────────────────┬─────────────────────────────┐
           │Path                    │ Description                 │
           ├────────────────────────┼─────────────────────────────┤
           │/etc/systemd/system     │ Local configuration         │
           ├────────────────────────┼─────────────────────────────┤
           │/run/systemd/system     │ Runtime units               │
           ├────────────────────────┼─────────────────────────────┤
           │/usr/lib/systemd/system │ Units of installed packages │
           └────────────────────────┴─────────────────────────────┘

When they say "installed packages" they're referring to anything which was installed via an RPM. The same can be assumed for Debian/Ubuntu as well where a DEB file would be the "installed package".

NOTE: the table above from a Debian/Ubuntu system is slightly different.

 Table 1.  Load path when running in system mode (--system).
       ┌────────────────────┬─────────────────────────────┐
       │Path                │ Description                 │
       ├────────────────────┼─────────────────────────────┤
       │/etc/systemd/system │ Local configuration         │
       ├────────────────────┼─────────────────────────────┤
       │/run/systemd/system │ Runtime units               │
       ├────────────────────┼─────────────────────────────┤
       │/lib/systemd/system │ Units of installed packages │
       └────────────────────┴─────────────────────────────┘

Analyzing /usr/lib/systemd/system

You can tell what packages own which unit files in /usr/lib/systemd/system like this on a CentOS/Fedora/RHEL system:

$ rpm -qf /usr/lib/systemd/system/* |sort -u | head
abrt-2.1.11-50.el7.centos.x86_64
abrt-addon-ccpp-2.1.11-50.el7.centos.x86_64
abrt-addon-kerneloops-2.1.11-50.el7.centos.x86_64
abrt-addon-pstoreoops-2.1.11-50.el7.centos.x86_64
abrt-addon-vmcore-2.1.11-50.el7.centos.x86_64
abrt-addon-xorg-2.1.11-50.el7.centos.x86_64
accountsservice-0.6.45-7.el7.x86_64
acpid-2.0.19-8.el7.x86_64
alsa-utils-1.1.3-2.el7.x86_64
anaconda-core-21.48.22.134-1.el7.centos.x86_64

Analyzing /etc/systemd/system

If we do the same against /etc/systemd/system, we'd expect to find no files owned by an RPM (Which is in fact the case on my CentOS 7.x system. ):

$ rpm -qf /etc/systemd/system/* /etc/systemd/system/*/* | grep -v 'not owned'
$

Outliers

Keep in mind that you may find occasional stray files under /usr/lib/systemd/system, such as with Virtualbox (vboxadd*):

$ rpm -qf /usr/lib/systemd/system/* |sort -u | grep 'not owned'
file /usr/lib/systemd/system/initrd.target.wants is not owned by any package
file /usr/lib/systemd/system/shutdown.target.wants is not owned by any package
file /usr/lib/systemd/system/vboxadd.service is not owned by any package
file /usr/lib/systemd/system/vboxadd-service.service is not owned by any package
file /usr/lib/systemd/system/vboxadd-x11.service is not owned by any package

There are others.

Conclusions

The expectation is that /usr/lib/systemd/system is a directory that should only contain systemd unit files which were put there by the package manager (YUM/DNF/RPM/APT/etc).

Files in /etc/systemd/system are manually placed here by the operator of the system for ad-hoc software installations that are not in the form of a package. This would include tarball type software installations or home grown scripts.

  • 3
    I was reluctant to click this google result because I was curious about /lib/systemd/system vs. /usr/lib/systemd/system. I'm glad I found this answer. – Bruno Bronosky Aug 6 '18 at 1:52
  • 1
    Placing a service definition in /etc/systemd/system generates an error if you mask it: Failed to execute operation: Invalid argument; systemd tries to replace the file with a symlink to /dev/null. Not saying this answer is incorrect, just something to remember. – Mrten Jan 22 at 17:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.