Linux - mount command returns zero/0 but not working

Question

Sounds strange, I have a shell script that it gets trigger by udev rules, to mount attached USB device to the system file tree. The script runs when an usb device attaches to system, so the rules seems to be fine. I monitor how the script progress by syslog, and it also goes fine, and even mount command returns zero, and it says:

 root[1023]: mount: /dev/sda1 mounted on /media/partitionlabel.

But at the end the device is not mounted, it is not listed in /etc/mtab - /proc/mounts - findmnt - mount. and if I run umount on device, it also says, the device is not mounted.

However If I run the script manually as a root from terminal, then it works perfect and device gets mount, but not when it runs by udev.

I've added 8 second sleep time to the start of the script, to make sure it's not a timing problem and also removed number from rules file name to make sure udevd would put the new rules at the bottom of rules queue, and script would run after other system rules, but no success.

The syslog:

(right after the device attached)

kernel: usb 1-1.2: new high-speed USB device number 12 using dwc_otg
kernel: usb 1-1.2: New USB device found, idVendor=058f, idProduct=6387
kernel: usb 1-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
kernel: usb 1-1.2: Product: Mass Storage
kernel: usb 1-1.2: Manufacturer: Generic
kernel: usb 1-1.2: SerialNumber: 24DCF568
kernel: usb-storage 1-1.2:1.0: USB Mass Storage device detected
kernel: scsi host6: usb-storage 1-1.2:1.0
kernel: scsi 6:0:0:0: Direct-Access     Generic  Flash Disk       8.07 PQ: 0 ANSI: 4
kernel: sd 6:0:0:0: [sda] 1968128 512-byte logical blocks: (1.00 GB/961 MiB)
kernel: sd 6:0:0:0: [sda] Write Protect is off
kernel: sd 6:0:0:0: [sda] Mode Sense: 23 00 00 00
kernel: sd 6:0:0:0: [sda] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
kernel:  sda: sda1
kernel: sda: p1 size 1968126 extends beyond EOD, enabling native capacity
kernel:  sda: sda1
kernel: sda: p1 size 1968126 extends beyond EOD, truncated
kernel: sd 6:0:0:0: [sda] Attached SCSI removable disk
root[1004]: /usr/local/sbin/udev-auto-mount.sh - status: started to automount sda1
root[1019]: /usr/local/sbin/udev-auto-mount.sh - status: Device Label is partitionlabel and Filesystem is vfat.
root[1021]: /usr/local/sbin/udev-auto-mount.sh - status: mounting the device sda1 by filesystem vfat to /media/partitionlabel.
root[1023]: mount: /dev/sda1 mounted on /media/partitionlabel.
root[1024]: /usr/local/sbin/udev-auto-mount.sh status: mount command proceed for vfat, retval is 0
root[1025]: /usr/local/sbin/udev-auto-mount.sh - status: succeed!

Configs:

/etc/udev/rules.d/local-rules:

The defined rule in udev is:

# /etc/udev/rules.d/local-rules
ENV{ID_BUS}=="usb",     ACTION=="add",  ENV{DEVTYPE}=="partition",      \
          RUN+="/usr/local/sbin/udev-automounter.sh %k $ENV{ID_FS_LABEL_ENC}"

udev-auto-mount.sh

The script starts by another script which defined in the udev rule. It so straight, it makes mount point directory and mounts the usb device to the mount point using its file system type and some regular options. I've added "-v" option to the mount command to be more verbose and also redirected all outputs to syslog, so I can see how it runs, but it not says too much.

#!/bin/sh
## /usr/local/sbin/udec-auto-mount.sh
##

logger -s "$0 - status: started to automount ${1}"
DEVICE=$1
sleep 8 

#...
#...
# Checking inputs, getting filesystem type (ID_FS_TYPE), partition label
# (ID_FS_LABEL) and ...

mkdir "/media/${ID_FS_LABEL}"


logger -s "$0 - status: mounting the device ${DEVICE} by filesystem ${ID_FS_TYPE} to /media/${ID_FS_LABEL}."
case $ID_FS_TYPE in
    vfat)   mount -v -t vfat -o sync,noatime,nosuid,nodev /dev/${DEVICE} "/media/${ID_FS_LABEL}" 2>&1 | logger
        let retVal=$?
        logger -s "$0 status: mount command proceed for vfat, retval is ${retVal}"
        ;;

    *)  mount -v -t auto -o sync,noatime /dev/${DEVICE} "/media/${ID_FS_LABEL}"
        ;;
esac
if [ ${retVal} -eq 0 ]; then
    logger -s "$0 - status: succeed!"
    exit 0
else
    logger -s "$0 Error: unable to mount the device ${DEVICE}, retval is ${retVal}"
    rmdir "/media/${ID_FS_LABEL}"
fi

exit 0

Maybe it helps:

Sometimes, after the script fails to mount the USB device, when I detach the device, some error come to syslog like:

kernel: usb 1-1.2: USB disconnect, device number 11
systemd-udevd[143]: error: /dev/sda: No such file or directory
systemd-udevd[977]: inotify_add_watch(7, /dev/sda, 10) failed: No such file or directory

Edit:

This is 'mount' version:

$ mount -V:
mount from util-linux 2.27.1 (libmount 2.27.0: assert, debug)

Answer 1

On a system with systemd, this problem can be encountered when you reformat the partition and try to mount it back.

I moved a disk from encryption to unencrypted, causing systemd’s generated mnt-disk.mount to (where mnt-disk is mount path from /etc/fstab) to refer the old path that didn’t exist any more, causing mount to go haywire.

Just doing systemctl daemon-reload and then doing the mount makes things work.

Answer 2

The sequence

mount ... | logger
rc=$?

isn't working as you expect: The return value of a pipe is the return value of the last element in that pipe.

$> false | true; echo $?
0

If you use bash, try PIPESTATUS:

$> false | true; echo $? ${PIPESTATUS[0]}
0 1
$> true | false; echo $? {PIPESTATUS[0]}
1 0

PIPESTATUS is a array variable. Details in the man page of bash. Other shells might have something similar to PIPESTATUS

Answer 3

Finally found the answer here.

Actually the problem were came from systemd-udevd which succeeded original udev. The systemd-udevd creates it's own mirror of root file system, when the 'udev' rule mount the device, it gets mounted and is accessible from:

/proc/{PID of systemd-udevd service}/root/{path to mount point}

but it is not visible from main root filesystem /.

archlinux's wiki (here) suggests:

Warning: To mount removable drives, do not call mount from udev rules. In case of FUSE filesystems, you will get Transport endpoint not connected errors. Instead, you could use udisks that handles automount correctly or to make mount work inside udev rules, copy /usr/lib/systemd/system/systemd-udevd.service to /etc/systemd/system/systemd-udevd.service and replace MountFlags=slave to MountFlags=shared.[3] Keep in mind though that udev is not intended to invoke long-running processes.

Solution:

I copied /usr/lib/systemd/system/systemd-udevd.service to the /etc/systemd/sytem/ directory and replaced MountFlags=slave by MountFlags=shared. then restarted the system, and now everything works fine.