Before all the unit files were in
/etc/systemd/system/ but now some are showing up in
/usr/lib/systemd/system (<- on CentOS, or
/lib/systemd/system <- on Debian/Ubuntu), what is the difference between these folders?
This question is already answered in
man 7 file-hierarchy which comes with systemd (there is also online version):
/etc System-specific configuration. (…) VENDOR-SUPPLIED OPERATING SYSTEM RESOURCES /usr Vendor-supplied operating system resources. Usually read-only, but this is not required. Possibly shared between multiple hosts. This directory should not be modified by the administrator, except when installing or removing vendor-supplied packages.
Basically, files that ships in packages downloaded from distribution repository go into
/usr/lib/systemd/. Modifications done by system administrator (user) go into
System-specific units override units supplied by vendors. Using drop-ins, you can override only specific parts of unit files, leaving the rest to vendor (drop-ins are available since the very beginning of systemd, but were properly documented only in v219; see
It seems at least sometimes one overwrite the other one. For e.g., I tweak
sudo systemctl daemon-reloadand changes are copied to
/lib/systemd/system/apache2.service(a Debian 9).– Pablo ANov 10, 2022 at 22:45
@PabloA Services in
/etc/systemd/system/multi-user.target.wants/are symbolic links to the real services.– MossmyrFeb 16 at 14:18
If you look at the man page
man systemd.unit it has a table that explains the differences. This is from a CentOS 7.x system.
UNIT LOAD PATH Unit files are loaded from a set of paths determined during compilation, described in the two tables below. Unit files found in directories listed earlier override files with the same name in directories lower in the list. Table 1. Load path when running in system mode (--system). ┌────────────────────────┬─────────────────────────────┐ │Path │ Description │ ├────────────────────────┼─────────────────────────────┤ │/etc/systemd/system │ Local configuration │ ├────────────────────────┼─────────────────────────────┤ │/run/systemd/system │ Runtime units │ ├────────────────────────┼─────────────────────────────┤ │/usr/lib/systemd/system │ Units of installed packages │ └────────────────────────┴─────────────────────────────┘
When they say "installed packages" they're referring to anything which was installed via an RPM. The same can be assumed for Debian/Ubuntu as well where a DEB file would be the "installed package".
NOTE: the table above from a Debian/Ubuntu system is slightly different.
Table 1. Load path when running in system mode (--system). ┌────────────────────┬─────────────────────────────┐ │Path │ Description │ ├────────────────────┼─────────────────────────────┤ │/etc/systemd/system │ Local configuration │ ├────────────────────┼─────────────────────────────┤ │/run/systemd/system │ Runtime units │ ├────────────────────┼─────────────────────────────┤ │/lib/systemd/system │ Units of installed packages │ └────────────────────┴─────────────────────────────┘
You can tell what packages own which unit files in
/usr/lib/systemd/system like this on a CentOS/Fedora/RHEL system:
$ rpm -qf /usr/lib/systemd/system/* |sort -u | head abrt-2.1.11-50.el7.centos.x86_64 abrt-addon-ccpp-2.1.11-50.el7.centos.x86_64 abrt-addon-kerneloops-2.1.11-50.el7.centos.x86_64 abrt-addon-pstoreoops-2.1.11-50.el7.centos.x86_64 abrt-addon-vmcore-2.1.11-50.el7.centos.x86_64 abrt-addon-xorg-2.1.11-50.el7.centos.x86_64 accountsservice-0.6.45-7.el7.x86_64 acpid-2.0.19-8.el7.x86_64 alsa-utils-1.1.3-2.el7.x86_64 anaconda-core-184.108.40.206-1.el7.centos.x86_64
If we do the same against
/etc/systemd/system, we'd expect to find no files owned by an RPM (Which is in fact the case on my CentOS 7.x system.
$ rpm -qf /etc/systemd/system/* /etc/systemd/system/*/* | grep -v 'not owned' $
Keep in mind that you may find occasional stray files under
/usr/lib/systemd/system, such as with Virtualbox (vboxadd*):
$ rpm -qf /usr/lib/systemd/system/* |sort -u | grep 'not owned' file /usr/lib/systemd/system/initrd.target.wants is not owned by any package file /usr/lib/systemd/system/shutdown.target.wants is not owned by any package file /usr/lib/systemd/system/vboxadd.service is not owned by any package file /usr/lib/systemd/system/vboxadd-service.service is not owned by any package file /usr/lib/systemd/system/vboxadd-x11.service is not owned by any package
There are others.
The expectation is that
/usr/lib/systemd/system is a directory that should only contain systemd unit files which were put there by the package manager (YUM/DNF/RPM/APT/etc).
/etc/systemd/system are manually placed here by the operator of the system for ad-hoc software installations that are not in the form of a package. This would include tarball type software installations or home grown scripts.
5I was reluctant to click this google result because I was curious about
/usr/lib/systemd/system. I'm glad I found this answer. Aug 6, 2018 at 1:52
1Placing a service definition in
/etc/systemd/systemgenerates an error if you mask it:
Failed to execute operation: Invalid argument; systemd tries to replace the file with a symlink to /dev/null. Not saying this answer is incorrect, just something to remember.– MrtenJan 22, 2019 at 17:59
@BrunoBronosky Debian actually uses both
/usr/lib/systemd/system, therefore I asked the question separately unix.stackexchange.com/questions/550001/…– pevikNov 2, 2019 at 6:57
This should be made the real answer. Thanks– user420792Jan 4, 2021 at 9:45
The runtime files in
/run/systemd/system stem from the ability to make modifications to a process (unit) during the current boot without that change/modification persisting across a reboot.
--runtime When used with enable, disable, edit, (and related commands), make changes only temporarily, so that they are lost on the next reboot. This will have the effect that changes are not made in subdirectories of /etc/ but in /run/, with identical immediate effects, however, since the latter is lost on reboot, the changes are lost too. Similarly, when used with set-property, make changes only temporarily, so that they are lost on the next reboot.
Debian systems you can use
dpkg-query -S to look for unit files with or without a package.