3

I'm trying to set-up a Raspberry Pi with read-only filesystems. The base image is the debian buster lite raspbian image.
Most stuff is working, e.g. had to do simple fixes like:

mv /etc/resolv.conf /var/run/resolv.conf && ln -s /var/run/resolv.conf /etc/resolv.conf

Similar for dhcp and a few other services

However, there is one service that I would like to get working, that refuses to work like this: systemd-timesync.

Here's what I did and what happens.systemd-timesync I created a directory /run/systemd-timesync, owned by user systemd-timesync:systemd-timesync Then created a symlink /var/lib/systemd/timesync -> /run/systemd-timesync

root@raspberrypi:/var/lib/systemd # ls -l /var/lib/systemd/timesync
lrwxrwxrwx 1 root root 21 Dec 25 14:48 /var/lib/systemd/timesync -> /run/systemd-timesync

root@raspberrypi:/var/lib/systemd # ls -l /run/systemd-timesync/
total 0
-rw-r--r-- 1 systemd-timesync systemd-timesync 0 Dec 25 15:02 clock

The relevant part of the systemd unit file:

...
[Service]
User=systemd-timesync
AmbientCapabilities=CAP_SYS_TIME
CapabilityBoundingSet=CAP_SYS_TIME
WorkingDirectory=/run/systemd-timesync
Environment=SYSTEMD_LOG_LEVEL=debug
ExecStartPre=/bin/pwd
ExecStart=!!/lib/systemd/systemd-timesyncd
...
RuntimeDirectory=systemd/timesync
StateDirectory=systemd/timesync
...

Note that I added a ExecStartPre=/bin/pwd which should just output the current working directory to the journal.

Now if I start the systemd-timesync with / mounted as read-only, this is what happens

root@raspberrypi:/var/lib/systemd # systemctl stop systemd-timesyncd.service && systemctl daemon-reload && systemctl start systemd-timesyncd.service
Job for systemd-timesyncd.service failed because of unavailable resources or another system error.
See "systemctl status systemd-timesyncd.service" and "journalctl -xe" for details.

The output of journalctl

Dec 25 15:34:10 raspberrypi systemd[1]: systemd-timesyncd.service: Trying to enqueue job systemd-timesyncd.service/stop/replace
Dec 25 15:34:10 raspberrypi systemd[1]: systemd-timesyncd.service: Installed new job systemd-timesyncd.service/stop as 1214
Dec 25 15:34:10 raspberrypi systemd[1]: systemd-timesyncd.service: Enqueued job systemd-timesyncd.service/stop as 1214
Dec 25 15:34:10 raspberrypi systemd[1]: systemd-timesyncd.service: Job 1214 systemd-timesyncd.service/stop finished, result=done
Dec 25 15:34:10 raspberrypi systemd[1]: /lib/systemd/system/systemd-timesyncd.service:36: Failed to parse system call, ignoring: io_pgetevents
Dec 25 15:34:10 raspberrypi systemd[1]: systemd-timesyncd.service: Changed dead -> failed
Dec 25 15:34:11 raspberrypi systemd[1]: systemd-timesyncd.service: Trying to enqueue job systemd-timesyncd.service/start/replace
Dec 25 15:34:11 raspberrypi systemd[1]: systemd-timesyncd.service: Installed new job systemd-timesyncd.service/start as 1215
Dec 25 15:34:11 raspberrypi systemd[1]: systemd-timesyncd.service: Enqueued job systemd-timesyncd.service/start as 1215
Dec 25 15:34:11 raspberrypi systemd[1]: systemd-timesyncd.service: ConditionFileIsExecutable=!/usr/sbin/VBoxService succeeded.
Dec 25 15:34:11 raspberrypi systemd[1]: systemd-timesyncd.service: ConditionFileIsExecutable=!/usr/sbin/chronyd succeeded.
Dec 25 15:34:11 raspberrypi systemd[1]: systemd-timesyncd.service: ConditionFileIsExecutable=!/usr/sbin/openntpd succeeded.
Dec 25 15:34:11 raspberrypi systemd[1]: systemd-timesyncd.service: ConditionFileIsExecutable=!/usr/sbin/ntpd succeeded.
Dec 25 15:34:11 raspberrypi systemd[1]: systemd-timesyncd.service: ConditionVirtualization=!container succeeded.
Dec 25 15:34:11 raspberrypi systemd[1]: systemd-timesyncd.service: ConditionCapability=CAP_SYS_TIME succeeded.
Dec 25 15:34:11 raspberrypi systemd[1]: systemd-timesyncd.service: Failed to run 'start-pre' task: Read-only file system
Dec 25 15:34:11 raspberrypi systemd[1]: systemd-timesyncd.service: Failed with result 'resources'.
Dec 25 15:34:11 raspberrypi systemd[1]: systemd-timesyncd.service: Job 1215 systemd-timesyncd.service/start finished, result=failed
Dec 25 15:34:11 raspberrypi systemd[1]: Failed to start Network Time Synchronization.

Clearly the /bin/pwd from ExecStartPre fails because of the read-only filesystem. I do not understand this, and don't know how to work around it. If I remove the ExecStartPre the same happens with the ExecStart command.

When I however do,

mount -o remount,rw /

and subsequently

root@raspberrypi:/var/lib/systemd # systemctl stop systemd-timesyncd.service && systemctl daemon-reload && systemctl start systemd-timesyncd.service

all works fine, including the pwd output to the journal.

Similarly when I start

root@raspberrypi:/var/lib/systemd # /lib/systemd/systemd-timesyncd
Synchronized to time server for the first time 84.199.86.247:123 (0.debian.pool.ntp.org).

all works fine.

So far, my conclusion seems to be that systemd REQUIRES write-access somewhere to start any command in ExecStartPre or ExecStart.

Any ideas on how I can achieve my original goal of having the raspberry update it's time settings?

Note: it may be related to the lines StateDirectory, RuntimeDirectory in the unit file.

4 Answers 4

1

That solved the problem on DietPi for me:

  • /run is mounted as tmpfs

  • Create a softlink for the StateDirectory under /var/lib:

    ln -s /run /var/lib/run

  • disable PrivateTmp

    #PrivateTmp=yes

  • changing StateDirectory to tmpfs:

    StateDirectory=run/timesync

RuntimeDirectory is absolute to /run, wich already is a tmpfs, so this doesn't need to be changed

1
  • This is more or less equivalent to StateDirectory=, because the directory used by systemd-timesyncd is hardcoded.
    – tc.
    Jun 19, 2021 at 21:48
0

After a long time of tinkering around, it turns out that commenting out the following 2 lines in the [Unit] section is enough:

#CapabilityBoundingSet=CAP_SYS_TIME
#PrivateTmp=yes

These turn out to be the only 2 lines that require commenting out, to have it all working on a read-only / filesystem.

It's beyond me why uncommenting any of these 2 lines breaks systemd-timesync working on a read-only filesystem. I would expect PrivateTmp=yes to create a private tmpfs mounted on the private /tmp for that process, but apparently not. May be a bug in systemd then.

0

You should consider to use overlayfs to make read only (but fake writable) the / filesystem. This will prevent some systemd services fails.

You can do this using raspi-config script in 4 Performance Options > P3 Overlay File System

Then the FS will be read only but fake writable (all changes will be lost at shutdown or reboot).

You can read the raspi-config script for further details.

0

Just setting StateDirectory= worked for me. Instead of editing system files, you can put a couple of lines in e.g. /etc/systemd/system/systemd-timesyncd.service.d/local-allow-readonly.conf:

[Service]
StateDirectory=

The directory/file path are hardcoded in timesyncd.c, so it doesn't change much, except User=systemd-timesync means that systemd-timesyncd can't create directory itself. So,

  • If you do this before systemd-timesync first runs (e.g. in an initial setup script), the directory (and clock file) will never be created.
  • If you do this after the directory has been created, the clock file can be updated when the filesystem is mounted read-write.

(If you're wondering where the error comes from, I'm guessing it's the attempt to chmod which fails with EROFS even if the mode is already set to the desired mode. It seems unrelated to PrivateTmp and CapabilityBoundingSet.)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.