Before all the unit files were in
/etc/systemd/system/ but now some are showing up in
/usr/lib/systemd/system (<- on CentOS, or
/lib/systemd/system <- on Debian/Ubuntu), what is the difference between these folders?
This question is already answered in
man 7 file-hierarchy which comes with systemd (there is also online version):
/etc System-specific configuration. (…) VENDOR-SUPPLIED OPERATING SYSTEM RESOURCES /usr Vendor-supplied operating system resources. Usually read-only, but this is not required. Possibly shared between multiple hosts. This directory should not be modified by the administrator, except when installing or removing vendor-supplied packages.
Basically, files that ships in packages downloaded from distribution repository go into
/usr/lib/systemd/. Modifications done by system administrator (user) go into
System-specific units override units supplied by vendors. Using drop-ins, you can override only specific parts of unit files, leaving the rest to vendor (drop-ins are available since the very beginning of systemd, but were properly documented only in v219; see
If you look at the man page
man systemd.unit it has a table that explains the differences. This is from a CentOS 7.x system.
UNIT LOAD PATH Unit files are loaded from a set of paths determined during compilation, described in the two tables below. Unit files found in directories listed earlier override files with the same name in directories lower in the list. Table 1. Load path when running in system mode (--system). ┌────────────────────────┬─────────────────────────────┐ │Path │ Description │ ├────────────────────────┼─────────────────────────────┤ │/etc/systemd/system │ Local configuration │ ├────────────────────────┼─────────────────────────────┤ │/run/systemd/system │ Runtime units │ ├────────────────────────┼─────────────────────────────┤ │/usr/lib/systemd/system │ Units of installed packages │ └────────────────────────┴─────────────────────────────┘
When they say "installed packages" they're referring to anything which was installed via an RPM. The same can be assumed for Debian/Ubuntu as well where a DEB file would be the "installed package".
NOTE: the table above from a Debian/Ubuntu system is slightly different.
Table 1. Load path when running in system mode (--system). ┌────────────────────┬─────────────────────────────┐ │Path │ Description │ ├────────────────────┼─────────────────────────────┤ │/etc/systemd/system │ Local configuration │ ├────────────────────┼─────────────────────────────┤ │/run/systemd/system │ Runtime units │ ├────────────────────┼─────────────────────────────┤ │/lib/systemd/system │ Units of installed packages │ └────────────────────┴─────────────────────────────┘
You can tell what packages own which unit files in
/usr/lib/systemd/system like this on a CentOS/Fedora/RHEL system:
$ rpm -qf /usr/lib/systemd/system/* |sort -u | head abrt-2.1.11-50.el7.centos.x86_64 abrt-addon-ccpp-2.1.11-50.el7.centos.x86_64 abrt-addon-kerneloops-2.1.11-50.el7.centos.x86_64 abrt-addon-pstoreoops-2.1.11-50.el7.centos.x86_64 abrt-addon-vmcore-2.1.11-50.el7.centos.x86_64 abrt-addon-xorg-2.1.11-50.el7.centos.x86_64 accountsservice-0.6.45-7.el7.x86_64 acpid-2.0.19-8.el7.x86_64 alsa-utils-1.1.3-2.el7.x86_64 anaconda-core-18.104.22.168-1.el7.centos.x86_64
If we do the same against
/etc/systemd/system, we'd expect to find no files owned by an RPM (Which is in fact the case on my CentOS 7.x system.
$ rpm -qf /etc/systemd/system/* /etc/systemd/system/*/* | grep -v 'not owned' $
Keep in mind that you may find occasional stray files under
/usr/lib/systemd/system, such as with Virtualbox (vboxadd*):
$ rpm -qf /usr/lib/systemd/system/* |sort -u | grep 'not owned' file /usr/lib/systemd/system/initrd.target.wants is not owned by any package file /usr/lib/systemd/system/shutdown.target.wants is not owned by any package file /usr/lib/systemd/system/vboxadd.service is not owned by any package file /usr/lib/systemd/system/vboxadd-service.service is not owned by any package file /usr/lib/systemd/system/vboxadd-x11.service is not owned by any package
There are others.
The expectation is that
/usr/lib/systemd/system is a directory that should only contain systemd unit files which were put there by the package manager (YUM/DNF/RPM/APT/etc).
/etc/systemd/system are manually placed here by the operator of the system for ad-hoc software installations that are not in the form of a package. This would include tarball type software installations or home grown scripts.