What's the difference between /usr/lib/systemd/system and /etc/systemd/system?

Question

Before all the unit files were in /etc/systemd/system/ but now some are showing up in /usr/lib/systemd/system (<- on CentOS, or /lib/systemd/system <- on Debian/Ubuntu), what is the difference between these folders?

Answer 1

This question is already answered in man 7 file-hierarchy which comes with systemd (there is also online version):

        /etc
           System-specific configuration.
 (…)
 VENDOR-SUPPLIED OPERATING SYSTEM RESOURCES
       /usr
            Vendor-supplied operating system resources. 
            Usually read-only, but this is not required. Possibly 
            shared between multiple hosts. This directory should not
            be modified by the administrator, except when installing 
            or removing vendor-supplied packages.

Basically, files that ships in packages downloaded from distribution repository go into /usr/lib/systemd/. Modifications done by system administrator (user) go into /etc/systemd/system/.

System-specific units override units supplied by vendors. Using drop-ins, you can override only specific parts of unit files, leaving the rest to vendor (drop-ins are available since the very beginning of systemd, but were properly documented only in v219; see man systemd.unit).

Answer 2

Background

If you look at the man page man systemd.unit it has a table that explains the differences. This is from a CentOS 7.x system.

   UNIT LOAD PATH
          Unit files are loaded from a set of paths determined during 
          compilation, described in the two tables below. Unit files found 
          in directories listed earlier override files with the same name 
          in directories lower in the list.

           Table 1.  Load path when running in system mode (--system).
           ┌────────────────────────┬─────────────────────────────┐
           │Path                    │ Description                 │
           ├────────────────────────┼─────────────────────────────┤
           │/etc/systemd/system     │ Local configuration         │
           ├────────────────────────┼─────────────────────────────┤
           │/run/systemd/system     │ Runtime units               │
           ├────────────────────────┼─────────────────────────────┤
           │/usr/lib/systemd/system │ Units of installed packages │
           └────────────────────────┴─────────────────────────────┘

When they say "installed packages" they're referring to anything which was installed via an RPM. The same can be assumed for Debian/Ubuntu as well where a DEB file would be the "installed package".

NOTE: the table above from a Debian/Ubuntu system is slightly different.

 Table 1.  Load path when running in system mode (--system).
       ┌────────────────────┬─────────────────────────────┐
       │Path                │ Description                 │
       ├────────────────────┼─────────────────────────────┤
       │/etc/systemd/system │ Local configuration         │
       ├────────────────────┼─────────────────────────────┤
       │/run/systemd/system │ Runtime units               │
       ├────────────────────┼─────────────────────────────┤
       │/lib/systemd/system │ Units of installed packages │
       └────────────────────┴─────────────────────────────┘

Analyzing /usr/lib/systemd/system

You can tell what packages own which unit files in /usr/lib/systemd/system like this on a CentOS/Fedora/RHEL system:

$ rpm -qf /usr/lib/systemd/system/* |sort -u | head
abrt-2.1.11-50.el7.centos.x86_64
abrt-addon-ccpp-2.1.11-50.el7.centos.x86_64
abrt-addon-kerneloops-2.1.11-50.el7.centos.x86_64
abrt-addon-pstoreoops-2.1.11-50.el7.centos.x86_64
abrt-addon-vmcore-2.1.11-50.el7.centos.x86_64
abrt-addon-xorg-2.1.11-50.el7.centos.x86_64
accountsservice-0.6.45-7.el7.x86_64
acpid-2.0.19-8.el7.x86_64
alsa-utils-1.1.3-2.el7.x86_64
anaconda-core-21.48.22.134-1.el7.centos.x86_64

Analyzing /etc/systemd/system

If we do the same against /etc/systemd/system, we'd expect to find no files owned by an RPM (Which is in fact the case on my CentOS 7.x system. ):

$ rpm -qf /etc/systemd/system/* /etc/systemd/system/*/* | grep -v 'not owned'
$

Outliers

Keep in mind that you may find occasional stray files under /usr/lib/systemd/system, such as with Virtualbox (vboxadd*):

$ rpm -qf /usr/lib/systemd/system/* |sort -u | grep 'not owned'
file /usr/lib/systemd/system/initrd.target.wants is not owned by any package
file /usr/lib/systemd/system/shutdown.target.wants is not owned by any package
file /usr/lib/systemd/system/vboxadd.service is not owned by any package
file /usr/lib/systemd/system/vboxadd-service.service is not owned by any package
file /usr/lib/systemd/system/vboxadd-x11.service is not owned by any package

There are others.

Conclusions

The expectation is that /usr/lib/systemd/system is a directory that should only contain systemd unit files which were put there by the package manager (YUM/DNF/RPM/APT/etc).

Files in /etc/systemd/system are manually placed here by the operator of the system for ad-hoc software installations that are not in the form of a package. This would include tarball type software installations or home grown scripts.

Answer 3

The runtime files in /run/systemd/system stem from the ability to make modifications to a process (unit) during the current boot without that change/modification persisting across a reboot.

From systemctl.1:

--runtime
           When used with enable, disable, edit, (and related commands),
           make changes only temporarily, so that they are lost on the
           next reboot. This will have the effect that changes are not
           made in subdirectories of /etc/ but in /run/, with identical
           immediate effects, however, since the latter is lost on
           reboot, the changes are lost too.

           Similarly, when used with set-property, make changes only
           temporarily, so that they are lost on the next reboot.

Debian systems you can use dpkg-query -S to look for unit files with or without a package.