4

I am working on a embedded Linux with u-boot as bootloader and systemd init system. The tools are limited to standard busybox.

While analyzing some problems, I figured out the root filesystem is read only which causes the problem. The reason is some of the services and programs depend on a writable root fs, causing malfunction.

After researching for a while I found out the root filesystem is only readonly when there is a power failure. (The main unit triggers a power recycle when some error is encountered.) I suspect that when the root fs tries to write to some critical file or updating some service / process, then on power failure a error flag is set on the filesystem. On next boot fsck reads the flag and mounts/remounts the root as read only, or fsck forces to go into some recovery mode (I don't know if any recovery mode exists).

Is my hypothesis correct? If so, then what is a fs flag that is set on the FS on error and how to I prevent the root to boot as RO?

Note:

  1. The root filesystem is mounted with 'errors=continue'. So if fsck reads the superblock for remount option, it should ignore the error and remounts as RW.

  2. I tried to reproduce the case, turning off the power while running a dd command, but never been able to reproduce.

Additional Question: Which udev/systemd magic mounts the root fs?

2

At boot, you are supposed to check your filesystems to see if the system was shut down properly or if it crashed, and perform the necessary recovery actions in the latter case. On modern journaled filesystems, this usually means a simple and quick journal recovery operation that can be done automatically.

Root filesystem checking and mounting is normally done by initramfs/initrd, but on an embedded system you might or might not have it.

If you are not using initramfs, then the traditional way would be to have the kernel always mount the root filesystem initially as read-only (with boot options root=/dev/<whatever> ro, and the start-up scripts would then first run fsck on it (assuming it's necessary for the filesystem type used) and then remount the root filesystem into read/write mode before doing anything else.

If initramfs did not check the root filesystem (perhaps because it's not being used), then the standard systemd service name for running a filesystem check on the root filesystem is named systemd-fsck-root.service. I could not find out the name of the service responsible for remounting the root filesystem with systemd after it's been checked.

If a boot-time root filesystem check needs to modify the root filesystem, it typically triggers another reboot afterwards, because the modification may have affected something the kernel has already read and is caching, and would now be inconsistent after a correction was made on the disk by fsck.

  • The answer is very helpful. The rootfs mount is handled by the kernel and I found systemd-remount-fs.service which probably remount the root fs. But how does the fsck decide that the filesystem has error or improperly powered off? Is there a flag that can be set to reproduce this event and test? One more suggestion I want is, how do I prevent the root fs to remain RO or how do I force remount as RW even if the shutdown was not proper, or there is a fs error flag... – dDebug Apr 26 '18 at 10:41
  • It depends on the filesystem type used. A filesystem metadata on the disc might record something like a timestamp of last mounting and last unmounting. If mount timestamp > unmount timestamp, the disk was not unmounted safely and would need checking. On journaled filesystems, if at mount time the journal area contains records that are not marked as committed to the actual filesystem, same thing. – telcoM Apr 26 '18 at 21:14
  • Basically, the checking for unsafe unmounting is done for a good reason. If you use a journaling filesystem, the necessary checking after a crash shutdown can be done completely automatically in at least about 99.9% of cases. But trying to routinely skip it and force mounting a filesystem that might be in an inconsistent state is a bad idea. – telcoM Apr 26 '18 at 21:19
  • Its is a journaled fs (ext3). But as far as I understand the fs superblock has a error flag to let fsck know what to do under a error condition, and if errors=continue, then the remount should be RW irrespective of the error, is it correct that I understand from the man page, also here unix.stackexchange.com/questions/97915/… – dDebug Apr 28 '18 at 4:17
  • That flag is for determining what to do when the filesystem is already mounted and an error is encountered. It is for when it is important that the system stays up, even if it might output corrupted data as a result. Conversely, if it's absolutely important that the system never outputs corrupted data, you can set errors=panic to make the system crash when it detects a disk error. But when you're mounting a filesystem, the situation is different: if the system crashed and rebooted, the damage is already done and you might as well take the time to recover properly. – telcoM Apr 28 '18 at 7:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.