3

My Fedora 27 x64 fails to boot after hard reset. It shows:

Failed to mount POSIX Message Queue File System,
Failed to start Remount and Kernel File Systems,
Failed to mount Kernel Debug File System,
Failed to mount Huge Pages File System [3]

and lots of other failures comes after these. See https://photos.app.goo.gl/qBUxT40zA2MTLTwO2

In all these cases

Failed at step EXEC spawning /usr/bin/mount: Permission denied

is given as a reason. How can it be? Doesn't it recognize it's own filesystems?

I have 3 kernels:

vmlinuz-4.14.16-300.fc27.x86_64
vmlinuz-4.15.13-300.fc27.x86_64
vmlinuz-4.15.14-300.fc27.x86_64

no matter which one I try to boot the same happens.

So far I have:

  1. Checked filesystem integrity with fsck. All partitions are clean.
  2. Checked disk health reported by SMART and performed both - short and long tests. Disk is perfectly healthy.
  3. Rebuilt initramfs. Mounted boot, proc, sys, dev in /mnt, chroot and sudo dracut.

Followed suggestions and:

  1. Performed fsck -f on /dev/mapper/fedora-home, got:

    tree extents for i-node 524820 (on level 2) could be narrower. Fix?<y>Y

Allowed to fix this.

And the same for /dev/mapper/fedora-root, /dev/sda1 (boot partition) confirmed they are clean. One more error of the same kind was found for an extra partition for data files.

  1. rpm -V --all | grep -v " [cg] " returned as follows:

.M.......    /run/libgpod
..5....T.    /var/lib/selinux/targeted/active/commit_num
.......T.    /var/lib/selinux/targeted/active/file_contexts
.......T.    /var/lib/selinux/targeted/active/homedir_template
S.5....T.    /var/lib/selinux/targeted/active/policy.kern
.M.....T.    /var/lib/selinux/targeted/active/seusers
.M.....T.    /var/lib/selinux/targeted/active/users_extra
.M.......    /var/run/pluto
not exists   /var/run/abrt
.M.......    /var/log/audit
not exists   /usr/lib/systemd/system-preset/85-display-manager.preset
S.5....T.    /usr/share/icons/Crux/icon-theme.cache
S.5....T.    /usr/share/icons/Mist/icon-theme.cache

  1. rpm -V "$(rpm -q --whatprovides /usr/bin/mount)"
    .M....G..  g /var/log/lastlog

  2. fixfiles check /usr

    libsemanage.semanage_make_sandbox: Error removing old sandbox directory /var/lib/selinux/targeted/tmp. (Read only file system). 
    genhomedircon: Could not begin transaction: Read only file system

Among many lines similar to to the one below:

Would relabel /usr/src/handbrake/trunk/build/contrib/lib from unconfined_u:object_r:usr_t:s0 to unconfined_u:object_r:lib_t:s0
a few interesting ones are:
Would relabel /usr/sbin/mount.nilfs2 from unconfined_u:object_r:bin_t:s0 to unconfined_u:object_r:mount_exec_t:s0
Would relabel /usr/sbin/umount.nilfs2 from unconfined_u:object_r:bin_t:s0 to unconfined_u:object_r:mount_exec_t:s0
Would relabel /usr/sbin/mkfs.nilfs2 from unconfined_u:object_r:bin_t:s0 to unconfined_u:object_r:fsadm_exec_t:s0

  1. Proved RAM works fine - memtest86 didn't find any errors during 3.5 passes and over 8h test time.

    9. Disabled SELinux (SELinux=disabled in /etc/selinux/config) and restarted. System started without any error! This proves problem is in SELinux policies. I believe I should start with checking those 6 top SELinux policies that have been changed somehow (see p. 5). The question is how to do it wisely.

  2. Checked local modifications to SELinux config files and file_contexts:

    semanage module -C -l
    Module name              Priority  Language
    semanage fcontext -C -l
    fcontext SELinux                                 type               Context
    /usr/bin/mount                                     all files          system_u:object_r:samba_share_t:s0
    /usr/share/dnfdaemon/dnfdaemon-system              all files          system_u:object_r:rpm_exec_t:s0
    /var/run/media/przemek/extra(/.*)?                 all files          system_u:object_r:samba_share_t:s0
    /var/www/html/photo                                all files          system_u:object_r:httpd_sys_rw_content_t:s0
    /var/www/html/photo/_cache                         all files          system_u:object_r:httpd_sys_rw_content_t:s0
    /var/www/html/photo/config                         all files          system_u:object_r:httpd_sys_rw_content_t:s0
    /var/www/html/photo/content                        all files          system_u:object_r:httpd_sys_rw_content_t:s0
    /var/www/html/photo/content/folders.json           all files          system_u:object_r:httpd_sys_rw_content_t:s0
    /var/www/html/photo/iv-config/language             all files          system_u:object_r:httpd_sys_rw_content_t:s0

Interestingly fcontext of the /usr/bin/mount has changed.

The system runs 24h/day as a simple home server (www, mail, etc.). From time to time (say once a few weeks) it freezes completely. HDD keeps writing something (repetitive, although irregular sound). No reaction to keyboard, mouse, remote SSH access. Many times I have tried to leave it overnight, but it does not recover, so I am forced to hard reset it each time this happens. This time I haven't waited, but hard reset it after just a few minutes. Unfortunately since then it cannot boot.

I remembered that a minute or less before the system froze Firefox message box appeared telling me that some script became irresponsive. I don't remember my choice (kill it/wait).

Hardware: Gigabyte GB-BACE-3160 Brix PC with Hitachi HTS725032A9A364 2.5" HDD and 4GB LPDDR3 RAM (default clock).

More details [here]

Failed to mount messages during boot Permission denied to mount POSIX Message File System, Kernel Debug File System Permission denied to mount Huge Pages File System, Remount Root and Kernel File Systems

  • 1
    Have you seen this one: forums.gentoo.org/viewtopic-t-1054286-start-0.html It could be a permissions/file ownership issue. It could also be bad RAM, have you tested RAM ? Another good read is linux-audit.com/… – thecarpy May 4 '18 at 10:34
  • Thank you for your comment. No, I haven't tested RAM yet. As to the ownership I will try to check it. – p.h. May 4 '18 at 14:25
  • ah, of course. Since system-remount-fs failed, your root filesystem is still mounted read-only. Since you are able to run mount manually, you can just mount / -oremount,rw. – sourcejedi May 4 '18 at 19:50
  • 1
    Since rpm -V (p. 5 above) didn't return any "U" or "G" flags set no files have changed user or group. Furthermore I couldn't find any files in /usr/bin, /usr/sbin, which were meant to be executable for root, but weren't. – p.h. May 5 '18 at 23:05
  • 1
    RAM is ok - checked with memtest (see p. 8). – p.h. May 6 '18 at 20:09
3

What causes permission to be denied for mounting rootfs, home, messeage queue, kernel file system, during boot?

[...]

In all these cases

Failed at step EXEC spawning /usr/bin/mount: Permission denied

is given as a reason. How can it be? Doesn't it recognize it's own filesystems?

The reason that it can't mount any of these filesystems, is literally that it cannot run the mount program due to a permission denied error. That's what the log messages you found are saying. We anticipate you will see the same error if you try to run mount yourself...

It sounds very much like the system on disk has been corrupted.

Generally, systems which freeze will not be happy, and this is not an entirely unexpected outcome. It is likely you are having a problem with your hardware, a kernel bug, or a hardware bug which the kernel currently does not know how to work around.

Faulty RAM can cause weird things, and it's relatively cheap to replace, so it's definitely worth looking at. You can use e.g. memtest86+ to test the RAM. Thanks to @thecarpy for suggesting this.

If Linux has a problem dealing some new but otherwise popular hardware, someone else might get the problem fixed in a newer version of Linux. If the hardware has been around for a while, or other Linux users don't use it, it might be very hard to fix.

I used one such system myself, affected by an issue like this one - which seems to have been an issue with very low power-draw Intel CPUs[1][2] - I think the J-series like your Celeron J3160 is somewhat higher power, so I'm not sure whether or not it could be the same issue.


However, if we ignore that entirely and wonder how to look at what the immediate cause of an "impossible" message like this might be, I have two more techniques to suggest. (And then a nitpick one of the techniques you already used, or at least how you described it to us).

  1. You can verify against original package checksums using rpm -V "$(rpm -q --whatprovides /usr/bin/mount)", and more widely using rpm -V --all. In the output, you should generally ignore lines which include " c " as these represent config files which are allowed to be changed. You will see a lot of these when you use --all. Also ignore lines which say " g "; this means dynamically "generated" files including some logfiles. So you might want to use rpm -V --all | grep -v " [cg] ". It's probably worth trying the first approach which just checks the package for mount to start with, because that will be a nice quick check. http://ftp.rpm.org/max-rpm/s1-rpm-verify-output.html
  2. Given that you are on Fedora, a second horror to check for regarding "Permission denied", would be corruption of SELinux attributes. The program for this is fixfiles. For example, fixfiles check. This too is likely to provide a horrifying amount of output which is not relevant, because not even Fedora developers actually know how to use SELinux correctly. I feel useful suggestions for this command are even harder to find. fixfiles check /usr should hopefully at least cover the specific error you have with /usr/bin/mount, while also looking for some wider damage, and avoiding the most noisy false positives.

So far I have:

  • Checked disk health reported by SMART and performed both - short and long tests. Disk is perfectly healthy.

Right, it's always good to test SMART health. And if you suspect disk corruption, running one of the tests (I assume long is best) is a good checklist item.

  • Checked filesystem integrity with fsck. All partitions are clean.

Good thought, but I have a nitpick. To specify that you have confirmed filesystem integrity, when faulty hardware appears to be a possibility, you must specify that you have run fsck -f.

(It is possible for the "dirty" state of a journalling filesystem such as ext4, to have been cleared by fsck or the kernel simply replaying the journal. In that case, running fsck without -f will simply report that the filesystem has been marked "clean", without checking any further.)

Also, you should specify that you're using ext4 for your root filesystem (assuming that's the case). If you use btrfs or maybe XFS, it's not necessarily the case that the traditional fsck command will achieve anything.

(In such cases filesystem-specific commands might be more helpful... or not. Basically, the ext2 line of filesystems originally pre-dates journalling and so developed a fsck which was intended to be able to do a thorough integrity check. Other filesystems have different histories.

btrfs and ZFS are "checksumming filesystems", written to detect errors on their own. It would be reasonable to consider running a "scrub" at this point. This would explicitly check data that has been written by the filesystem against the checksum that was generated at the time. (This excludes certain types of files which opt out, for performance reasons).

For example, if data was corrupted in the process of being transmitted to the disk, a "scrub" would detect this, but a SMART test would not.)

  • Thank you for this detailed answer! I will try to follow your suggestions and let you know. – p.h. May 4 '18 at 14:22
  • At the moment I have tried to mount -t mqueue none /dev/mqueue in emergency mode and it returned no error, but /dev/mqueue folder is empty. – p.h. May 4 '18 at 14:32
  • @p.h. ok. It's fine if that FS is empty, AFAIK. This shows your manual command was not affected by the error "spawning /usr/bin/mount: Permission denied". This surprises me. I suspect it will be hard to really debug that specific Permission denied error, so the broad-spectrum (but slow) checks might be more productive. SELinux is maybe slightly elevated as a candidate though. – sourcejedi May 4 '18 at 14:55
  • I have followed your advices. Please take a look at points 4-7. Surprisingly fixfiles says file system is read only and that it would like to relabel /usr/sbin/mount.nilfs2. – p.h. May 4 '18 at 19:11
  • SELinux is causing the problem as I have proved - see p. 9. How to examine why wisely? – p.h. May 6 '18 at 20:25
1

The problem was caused by improper file context of the /usr/bin/mount file: samba_share_t.

The file context change wasn't caused by some error due to hard reset, but... by my imprudent decision to follow the first suggestion of SELinux Alert Browser. See the screenshot below. enter image description here

This first suggestion was to change /usr/bin/mount file context to samba_share_t to allow smbd to access getattr.

The solution was:

  1. to delete invalid file context, restore default and relabel the file:
    [root@atlas ~]# ls -Z /usr/bin/mount
    system_u:object_r:samba_share_t:s0 /usr/bin/mount
    [root@atlas ~]# semanage fcontext -d /usr/bin/mount
    [root@atlas ~]# restorecon -v /usr/bin/mount
    Relabeled /usr/bin/mount from system_u:object_r:samba_share_t:s0 to system_u:object_r:mount_exec_t:s0
    [root@atlas ~]# ls -Z /usr/bin/mount
    system_u:object_r:mount_exec_t:s0 /usr/bin/mount
    
  2. reboot system.

It could be done in emergency console, but I have used the console to put SELinux into permissive mode, boot system and then change the file context as described above.

When I have checked modified SELinux contexts of files (see p. 10 of my initial post) I have noticed that context of the mount looked suspicious. At the moment I realized that shortly before the problem started I have imprudently followed the first suggestion of SELinux Alert Browser to change mount file context. The same suggestion appeared now, after system repair and restart, so I was able to attach the screenshot below.

Credit for @sourcejedi for pointing SELinux may be causing the problem and for his kind help!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.