5

I recently upgraded systemd from version 208 to 212 (also tried any other version between these two numbers)

And the system can't boot anymore. Looks like it cannot mount /tmp, but in emergency shell mount /tmp just works fine, so does every other file system. (including a LUKS volume)

Pretty sure this is not about initramfs or kernel modules, if I simply downgrade systemd again, everything is just fine. Also, mkinitcpio -p linux-custom doesn't help either

UPDATE

Looks like dbus connection is broken, when I try to run systemd-analyze blame it says failed to get dbus connection or something

EDIT

Other method I tried:

  1. Disable plymouth target (not even present in systemctl list-units)

  2. Disable the tmp.mount target, does not help either

Logs

I'm running Arch Linux, here's the fstab structure about tmpfs:

/dev/zram0 /tmp tmpfs       nodev,size=512M,nosuid,noatime,nodiratime   0       0
/dev/zram0 /var/tmp     tmpfs   nodev,size=256M,nosuid,noexec,noatime,nodiratime        0       0

Here's the attached journal log (only systemd part)

systemd-journal[198]: Runtime journal is using 8.0M (max allowed 388.0M, trying to leave 582.1M free of 3.7G available → current limit 388.0M).
systemd-journal[198]: Runtime journal is using 8.0M (max allowed 388.0M, trying to leave 582.1M free of 3.7G available → current limit 388.0M).
systemd-udevd[148]: starting version 212
systemd[1]: RTC configured in localtime, applying delta of 480 minutes to system time.
systemd 212 running in system mode. (+PAM -AUDIT -SELINUX -IMA -SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ +SECCOMP -APPARMOR)
systemd[1]: Detected architecture 'x86-64'.
systemd[1]: Failed to insert module 'ipv6'
systemd[1]: No hostname configured.
systemd[1]: Set hostname to <localhost>.
systemd[1]: Cannot add dependency job for unit display-manager.service, ignoring: Unit display-manager.service failed to load: No such file or directory.
systemd[1]: Starting Forward Password Requests to Wall Directory Watch.
systemd[1]: Started Forward Password Requests to Wall Directory Watch.
systemd[1]: Starting Remote File Systems.
systemd[1]: Reached target Remote File Systems.
systemd[1]: Starting Arbitrary Executable File Formats File System Automount Point.
systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point.
systemd[1]: Starting Dispatch Password Requests to Console Directory Watch.
systemd[1]: Started Dispatch Password Requests to Console Directory Watch.
systemd[1]: Starting Paths.
systemd[1]: Reached target Paths.
systemd[1]: Starting Swap.
systemd[1]: Reached target Swap.
systemd[1]: Expecting device dev-zram0.device...
systemd[1]: Expecting device dev-disk-by\x2duuid-fc660063\x2d302d\x2d4c49\x2d9a69\x2d3ffcf9217590.device...
systemd[1]: Expecting device dev-sdb3.device...
systemd[1]: Expecting device dev-sdb5.device...
systemd[1]: Expecting device dev-sdb7.device...
systemd[1]: Expecting device dev-mapper-secure.device...
systemd[1]: Expecting device dev-sda2.device...
systemd[1]: Expecting device dev-sda1.device...
systemd[1]: Starting Root Slice.
systemd[1]: Created slice Root Slice.
systemd[1]: Starting User and Session Slice.
systemd[1]: Created slice User and Session Slice.
systemd[1]: Starting Device-mapper event daemon FIFOs.
systemd[1]: Listening on Device-mapper event daemon FIFOs.
systemd[1]: Starting Delayed Shutdown Socket.
systemd[1]: Listening on Delayed Shutdown Socket.
systemd[1]: Starting /dev/initctl Compatibility Named Pipe.
systemd[1]: Listening on /dev/initctl Compatibility Named Pipe.
systemd[1]: Starting Syslog Socket.
systemd[1]: Listening on Syslog Socket.
systemd[1]: Starting udev Control Socket.
systemd[1]: Listening on udev Control Socket.
systemd[1]: Starting udev Kernel Socket.
systemd[1]: Listening on udev Kernel Socket.
systemd[1]: Starting LVM2 metadata daemon socket.
systemd[1]: Listening on LVM2 metadata daemon socket.
systemd[1]: Starting Journal Socket.
systemd[1]: Listening on Journal Socket.
systemd[1]: Starting System Slice.
systemd[1]: Created slice System Slice.
systemd[1]: Starting File System Check on Root Device...
systemd[1]: Starting system-getty.slice.
systemd[1]: Created slice system-getty.slice.
systemd[1]: Starting Apply Kernel Variables...
systemd[1]: Starting Load Kernel Modules...
systemd[1]: Mounting Huge Pages File System...
systemd[1]: Mounting POSIX Message Queue File System...
systemd[1]: Starting Setup Virtual Console...
systemd[1]: Starting udev Coldplug all Devices...
systemd\x2dcryptsetup.slice.
systemd\x2dcryptsetup.slice.
systemd[1]: Starting Set Up Additional Binary Formats...
systemd[1]: Mounted Debug File System.
systemd[1]: Starting Create list of required static device nodes for the current kernel...
systemd[1]: Starting Journal Service...
systemd[1]: Started Journal Service.
systemd-journal[198]: Journal started
systemd-modules-load[191]: Module 'acpi_cpufreq' is builtin
systemd-modules-load[191]: Module 'acpi_call' is builtin
systemd-modules-load[191]: Module 'aesni_intel' is builtin
systemd-modules-load[191]: Module 'coretemp' is builtin
systemd-modules-load[191]: Module 'microcode' is builtin
systemd-modules-load[191]: Module 'tp_smapi' is builtin
systemd-modules-load[191]: Module 'uinput' is builtin
systemd-sysctl[189]: Overwriting earlier assignment of kernel/sysrq in file '/etc/sysctl.d/99-sysctl.conf'.
systemd[1]: Starting Slices.
systemd[1]: Reached target Slices.
systemd-binfmt)
systemd-modules-load[191]: Inserted module 'vboxdrv'
systemd[1]: Mounted POSIX Message Queue File System.
systemd[1]: Mounted Arbitrary Executable File Formats File System.
systemd[1]: Started udev Coldplug all Devices.
systemd-modules-load[191]: Inserted module 'vboxnetadp'
systemd-modules-load[191]: Inserted module 'vboxnetflt'
systemd[1]: Started Set Up Additional Binary Formats.
systemd-modules-load[191]: Inserted module 'vboxpci'
systemd-modules-load[191]: Module 'zram' is builtin
systemd-modules-load[191]: Module 'fuse' is builtin
systemd[1]: Started Load Kernel Modules.
systemd[1]: Mounting Configuration File System...
systemd[1]: Mounting FUSE Control File System...
systemd[1]: Starting Create Static Device Nodes in /dev...
systemd[1]: Mounted FUSE Control File System.
systemd[1]: Mounted Configuration File System.
systemd[1]: Started Apply Kernel Variables.
systemd-fsck[188]: ArchRoot: clean, 281073/812800 files, 1963679/3249138 blocks
systemd[1]: Started File System Check on Root Device.
systemd[1]: Starting Remount Root and Kernel File Systems...
systemd[1]: Started Remount Root and Kernel File Systems.
systemd[1]: Starting Load/Save Random Seed...
systemd[1]: Started Load/Save Random Seed.
systemd[1]: Started Setup Virtual Console.
systemd[1]: Started Create Static Device Nodes in /dev.
systemd[1]: Starting udev Kernel Device Manager...
systemd[1]: Starting Local File Systems (Pre).
systemd[1]: Reached target Local File Systems (Pre).
systemd[1]: Mounting /run/shm...
systemd[1]: Mounted /run/shm.
systemd[1]: Started udev Kernel Device Manager.
systemd-udevd[234]: starting version 212
systemd-sysctl[253]: Overwriting earlier assignment of kernel/sysrq in file '/etc/sysctl.d/99-sysctl.conf'.
systemd-sysctl[295]: Overwriting earlier assignment of kernel/sysrq in file '/etc/sysctl.d/99-sysctl.conf'.
systemd-udevd[239]: error opening ATTR{/sys/devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.3/1-1.3:1.0/power/control} for writing: No such file or directory
systemd-journal[198]: Forwarding to syslog missed 40 messages.
systemd[1]: Job dev-sda1.device/start timed out.
systemd[1]: Timed out waiting for device dev-sda1.device.
systemd[1]: Dependency failed for /data.
systemd[1]: Dependency failed for Local File Systems.
systemd[1]: Triggering OnFailure= dependencies of local-fs.target.
systemd[1]: Job dev-sda2.device/start timed out.
systemd[1]: Timed out waiting for device dev-sda2.device.
systemd[1]: Dependency failed for /vm.
systemd[1]: Job dev-sdb7.device/start timed out.
systemd[1]: Timed out waiting for device dev-sdb7.device.
systemd[1]: Dependency failed for Cryptography Setup for secure.
systemd[1]: Dependency failed for Encrypted Volumes.
systemd[1]: Dependency failed for dev-mapper-secure.device.
systemd[1]: Dependency failed for /secure.
systemd[1]: Job dev-sdb5.device/start timed out.
systemd[1]: Timed out waiting for device dev-sdb5.device.
systemd[1]: Dependency failed for /ubuntu_ssd_vm.
systemd[1]: Job dev-sdb3.device/start timed out.
systemd[1]: Timed out waiting for device dev-sdb3.device.
systemd[1]: Dependency failed for /win7_ssd_vm.
systemd[1]: Job dev-disk-by\x2duuid-fc660063\x2d302d\x2d4c49\x2d9a69\x2d3ffcf9217590.device/start timed out.
systemd[1]: Timed out waiting for device dev-disk-by\x2duuid-fc660063\x2d302d\x2d4c49\x2d9a69\x2d3ffcf9217590.device.
systemd[1]: Dependency failed for /Application.
systemd[1]: Job dev-zram0.device/start timed out.
systemd[1]: Timed out waiting for device dev-zram0.device.
systemd[1]: Dependency failed for /tmp.
systemd[1]: Dependency failed for /var/tmp.
systemd[1]: Dependency failed for /home/USER/.cache.
systemd[1]: Dependency failed for /var/log.
systemd[1]: Dependency failed for Update UTMP about System Boot/Shutdown.
systemd[1]: Requested transaction contradicts existing jobs: Transaction is destructive.
systemd[1]: syslog.socket failed to queue service startup job (Maybe the service file is missing or not a non-template unit?): Transaction is destructive.
systemd[1]: Unit syslog.socket entered failed state.
systemd[1]: Stopped Daily Cleanup of Temporary Directories.
systemd[1]: Stopped Getty on tty1.
systemd[1]: Stopped Getty on tty3.
systemd[1]: Stopped target Graphical Interface.
systemd[1]: Stopped target Multi-User System.
systemd[1]: Stopped Initialize hardware monitoring sensors.
systemd[1]: Stopped Set brightness.
systemd[1]: Stopped Restore pax attributes.
systemd[1]: Stopped sets battery thresholds.
systemd[1]: Stopped Name Service Cache Daemon.
systemd[1]: Stopped Periodic Command Scheduler.
systemd[1]: Stopped Network Manager.
systemd[1]: Stopped My GoAgent Service.
systemd[1]: Starting Network.
systemd[1]: Reached target Network.
systemd[1]: Stopped Run my.local script.
systemd[1]: Stopped Load apparmor profiles.
systemd[1]: Stopped Login Service.
systemd[1]: Starting Login Prompts.
systemd[1]: Reached target Login Prompts.
systemd[1]: Stopped ACPI event daemon.
systemd[1]: Closed ACPID Listen Socket.
systemd[1]: Stopped D-Bus System Message Bus.
systemd[1]: Closed D-Bus System Message Bus Socket.
systemd[1]: Stopped System Logger Daemon.
systemd[1]: Stopped Permit User Sessions.
systemd[1]: Starting Trigger Flushing of Journal to Persistent Storage...
systemd[1]: Stopped target Basic System.
systemd[1]: Starting Timers.
systemd-journal[198]: Forwarding to syslog missed 26 messages.
systemd[1]: Reached target Timers.
systemd[1]: Starting Sockets.
systemd[1]: Reached target Sockets.
systemd[1]: Stopped target System Initialization.
systemd[1]: Started Manage Sound Card State (restore and store).
systemd[1]: Starting Restore Sound Card State...
systemd[1]: Starting Create Volatile Files and Directories...
systemd[1]: Starting Emergency Shell...
systemd[1]: Started Emergency Shell.
systemd[1]: Starting Emergency Mode.
systemd[1]: Reached target Emergency Mode.
systemd[1]: Started Restore Sound Card State.
systemd-journal[198]: Runtime journal is using 8.0M (max allowed 388.0M, trying to leave 582.1M free of 3.7G available → current limit 388.0M).
systemd[1]: Started Trigger Flushing of Journal to Persistent Storage.
systemd[1]: Started Create Volatile Files and Directories.
systemd[1]: Startup finished in 2.835s (kernel) + 1min 30.235s (userspace) = 1min 33.070s.
systemd[884]: Failed at step EXEC spawning /bin/plymouth: No such file or directory
  • uh... did you mean mkinitcpio? – strugee Apr 7 '14 at 4:37
  • @strugee yeah typo, fixed – daisy Apr 7 '14 at 4:38
6

Just connected with the developers, and my problem is resolved.

My customized kernel does not support the open by file handle syscall, so I enabled the CONFIG_FHANDLE option, and it boots again.

  • You had to recompile? You should accept this - obviously, its the right answer. – mikeserv Apr 7 '14 at 11:01
  • @mikeserv Yes I have fully automatic way to do that ;-P already recompiled and confirmed – daisy Apr 7 '14 at 11:02
  • Is this option turned on by default? – Piotr Dobrogost Nov 24 '14 at 15:52
3

Your disks aren't mounting - and I notice they're supposed to be encrypted. I wonder - what are their ownership settings?

208 Changelog:

    * A new kernel command line option luks.options= is understood
      now which allows specifiying LUKS options for usage for LUKS
      encrypted partitions specified with luks.uuid=.

Maybe it's expected you should be using that.

Though there were many other changes - including some very specific to the issue Goldilocks mentions. You should be looking through the changelog if you intend to use that version.

AND the 209 Changelog IS HUGE. This pertains specifically to disk encryption:

    * When parsing /etc/crypttab, support for a new key-slot=
      option as supported by Debian is added. It allows indicating
      which LUKS slot to use on disk, speeding up key loading.

Looking closely at your output, and I would suggest you switch from using 'sda...b...c' and etc and specify disk labels. You can have a look at your disks to verify their labels with lsblk: ( I mistakenly suspected you were booting from a USB device. Thank you for the correction)

    lsblk -f

NAME   FSTYPE LABEL     UUID                                 MOUNTPOINT
sda
├─sda1 vfat   ESP       F0B7-5DAE                            /esp
├─sda2 btrfs  btrfs_ssd c01f596a-3b39-4be6-b1e5-8ddf0d4685ff /
├─sda3 ext4             c79dbc36-923a-4bb1-ab6c-606ef7c1e1f6
├─sda4 ext4   tmp       a2d977e7-4025-4c7c-b161-49f4ec3697c9
└─sda5 ext4             36ee1d08-b085-4bce-bb0d-20b6d5b7268f
sdb    bcache           4bd63488-e1d7-4858-8c70-a35a5ba2c452
sdc    bcache           ce6de517-7538-45d6-b8c4-8546f13f76c1
sr0

How you set the labels depends on your filesystem, but you can specify them in /etc/fstab with LABEL=${label}.

You can also include an /etc/fstab specifically for early userspace by merely copying one into your initramfs as /etc/fstab.

And speaking of /etc/fstab this is also in the 209 changelog:

    * The FsckPassNo= compatibility option in mount/service units
      has been removed. The fstab generator will now add the
      necessary dependencies automatically, and does not require
      PID1's support for that anymore.

It's probably a good idea to verify your disk-encryption mount units are up-to-date. Don't rely on the aur for that. Any problem related to the above would result in fsck not passing for encrypted disks and they wouldn't be automatically mounted. This would not affect whether or not you could mount them on your own.

Here's another 209 changelog excerpt related to /tmp:

    * tmpfiles gained a new "--boot" option. When this is not used,
      only lines where the command character is not suffixed with
      "!" are executed. When this option is specified, those
      options are executed too. This partitions tmpfiles
      directives into those that can be safely executed at any
      time, and those which should be run only at boot (for
      example, a line that creates /run/nologin).

And by the way, if your problem is that you have to mount disks on your own in the recovery console, then your problem is that they're not mounting in initramfs. In other words, your initramfs image is your problem.

  • No I'm not booting from USB, just two SSDs – daisy Apr 7 '14 at 8:09
  • I mount the disk manually, including the LUKS one, then I did a systemctl default and it stuck – daisy Apr 7 '14 at 8:10
  • @war0ck Im sorry i thought i saw USB in your output there somewhere - I'll fix that, but you should use disk labels. – mikeserv Apr 7 '14 at 8:11
  • It stuck as in - it's fixed? Good - I bet it correctly generated those fstab.generator mount units. – mikeserv Apr 7 '14 at 8:15
  • I would guess "it stuck" as in "it got stuck", not "it's fixed" ;-) – Chris Down Apr 7 '14 at 8:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.