Why does a systemd unit file at `/usr/lib/systemd/system/` still works for Ubuntu?

Question

I understand that systemd stores unit files at different locations for different versions of Linux. On RHEL, it's at /usr/lib/systemd/system/, whereas on Debian-based machines it's at /lib/systemd/system/.

However, on my Ubuntu 18.04 machine, I just installed Elasticsearch using a .deb file, and its systemd unit file was installed under /usr/lib/systemd/system/, but systemd is still able to pick it up.

$ uname -a
Linux nucleolus 4.15.0-46-generic #49-Ubuntu SMP Wed Feb 6 09:33:07 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

$ sudo systemctl status elasticsearch.service
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: http://www.elastic.co

Note the path is /usr/lib/systemd/system/elasticsearch.service.

So why does a systemd unit file at /usr/lib/systemd/system/ still works for Ubuntu? What is the real unit file load path for Debian/Ubuntu systems?

Answer 1

The paths systemd looks up for unit files is read from UnitPath and can be queried with systemctl.

# systemctl --no-pager --property=UnitPath show | tr ' ' '\n'
UnitPath=/etc/systemd/system.control
/run/systemd/system.control
/run/systemd/transient
/etc/systemd/system
/run/systemd/system
/run/systemd/generator
/lib/systemd/system
/run/systemd/generator.late

As you can see, this does not include /usr/lib/systemd/system, which is the output on a Ubuntu 18.04 system. The UnitPath is generated during runtime and only directories that actually exist are shown here.

# mkdir -p /usr/lib/systemd/system
# systemctl daemon-reload
# systemctl --no-pager --property=UnitPath show | tr ' ' '\n' | grep "/usr/lib/systemd/system"
/usr/lib/systemd/system

So creating the directory was enough to add /usr/lib/systemd/system to UnitPath, which was likely done by installing Elasticsearch.

---

Which directories are taken into account when constructing UnitPath can be queried with pkg-config and the variables systemdsystemunitdir and systemdsystemunitpath.

# pkg-config systemd --variable=systemdsystemunitdir 
/lib/systemd/system

# pkg-config systemd --variable=systemdsystemunitpath | tr ':' '\n'
/etc/systemd/system
/etc/systemd/system
/run/systemd/system
/usr/local/lib/systemd/system
/lib/systemd/system
/usr/lib/systemd/system
/lib/systemd/system

In src/core/systemd.pc.in the systemdsystemunitpath is as follows.

systemdsystemunitpath=${systemdsystemconfdir}:/etc/systemd/system:/run/systemd/system:/usr/local/lib/systemd/system:${systemdsystemunitdir}:/usr/lib/systemd/system:/lib/systemd/system