Difference between /usr/lib/systemd/*/*.service and /lib/systemd/*/*.service

Question

On Debian some systemd services are installed to /usr/lib/systemd/*/*.service, e.g.:

/usr/lib/systemd/user/org.gnome.Evince.service
/usr/lib/systemd/user/pulseaudio.service
/usr/lib/systemd/user/gpg-agent.service

Other services (actually much more) are in /lib/systemd/*/*.service

/lib/systemd/system/networking.service
/lib/systemd/system/sddm.service
/lib/systemd/system/apache2.service
/lib/systemd/system/dm-event.service
/lib/systemd/system/ModemManager.service

There are folders in /lib/systemd:

/lib/systemd/network/
/lib/systemd/system/
/lib/systemd/system-generators/
/lib/systemd/system-preset/
/lib/systemd/system-shutdown/
/lib/systemd/system-sleep/

/usr/lib/systemd:

/usr/lib/systemd/boot/
/usr/lib/systemd/catalog/
/usr/lib/systemd/scripts/
/usr/lib/systemd/system/
/usr/lib/systemd/user/
/usr/lib/systemd/user-environment-generators/
/usr/lib/systemd/user-generators/
/usr/lib/systemd/user-preset/

So, what is the difference of these two directories? Systemd documentation does not mention /lib/systemd at all.

Is it the place chosen by distro or software upstream? E.g. for Apache Debian uses /lib/systemd/system/apache2.service, but buildroot /usr/lib/systemd/system/apache.service (it looks like buildroot also uses /lib/systemd at all).

Is it somehow affected by /usr merge?

Everybody cares just about difference between /usr/lib/systemd/system and /etc/systemd/system (it touches the topic showing that path for Units of installed packages is distro specific – Centos 7 uses /usr/lib/systemd/system, Debian /lib/systemd/system, but Debian uses both paths).

Answer 1

Sources answer the question, it's really affected by /usr merge, see LOOKUP_PATHS_SPLIT_USR:

src/portable/portable.c: we force looking inside of /lib/systemd/system/ for units too, as we might be compiled for a split-usr system but the image might be a legacy-usr one

/* Then, send unit file data to the parent (or/and add it to the hashmap).
 * For that we use our usual unit discovery logic. Note that we force looking 
 * inside of /lib/systemd/system/ for units too, as we might be
 * compiled for a split-usr system but the image might be a legacy-usr one. */
r = lookup_paths_init(&paths, UNIT_FILE_SYSTEM, LOOKUP_PATHS_SPLIT_USR, where);

(comment reformatted for better readability)

src/shared/path-lookup.c: add "/lib/systemd/system" if flag LOOKUP_PATHS_SPLIT_USR

case UNIT_FILE_SYSTEM:
   add = strv_new(
      /* If you modify this you also want to modify
       * systemdsystemunitpath= in systemd.pc.in! */
      ...
      "/usr/local/lib/systemd/system",
      SYSTEM_DATA_UNIT_PATH,
      "/usr/lib/systemd/system",
      STRV_IFNOTNULL(flags & LOOKUP_PATHS_SPLIT_USR ? "/lib/systemd/system" : NULL),
            ...

src/core/systemd.pc.in: /usr/lib/systemd/system and /lib/systemd/system

systemdsystemunitpath=${systemdsystemconfdir}:/etc/systemd/system:\
/run/systemd/system\:/usr/local/lib/systemd/system:${systemdsystemunitdir}\:
/usr/lib/systemd/system:/lib/systemd/system

(reformatted for better readability)

Commit message from 799b210267 ("path-lookup: add flag to optionally force checking split-usr unit dirs"):

When we look into a portable service image it might contain the unit
files in split-usr directories rather than merged-usr directories as on
the host. Hence, let#s add a flag that checking all dirs can be forced.