0

i have a VM machine that rebooted last night and we could't connect to it via ssh. i used console and saw that only / and swap are mounted with lsblk command:

  NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
  sr0     11:0    1 1024M  0 rom
  sdb      8:16   0    8G  0 disk
  ├─sdb1   8:17   0    2G  0 part 
  └─sdb2   8:18   0    6G  0 part [SWAP]
  sdc      8:32   0   20G  0 disk
  └─sdc1   8:33   0   20G  0 part 
  sde      8:64   0  400M  0 disk
  └─sde1   8:65   0  399M  0 part 
  sda      8:0    0   20G  0 disk
  └─sda1   8:1    0   20G  0 part /
 sdd      8:48   0   20G  0 disk
 └─sdd1   8:49   0   20G  0 part 
 sdf      8:80   0   10G  0 disk
 └─sdf1   8:81   0   10G  0 part   

but when i ran df -h:

  Filesystem      Size  Used Avail Use% Mounted on
  /dev/sda1        20G  1.2G   18G   7% /
  tmpfs           7.8G     0   7.8G   0% /dev/shm
  /dev/sde1        20G    1.2G  18G   7% /boot
  /dev/sdd1        20G    1.2G  18G   7% /data
  /dev/sdc1        20G    1.2G  18G   7% /opt
  /dev/sdb1        20G    1.2G  18G   7% /var
  /dev/sdf1        20G    1.2G  18G   7% /backup

when i ran ls -hal on them they were all empty except for /.i tried to unmount and mount partitions i got error that partitions are not mount. i mounted them all again for example:

mount /dev/sdf1 /backup

and df -h:

 Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        20G  1.2G   18G   7% /
tmpfs           7.8G     0  7.8G   0% /dev/shm
/dev/sde1       387M   40M  327M  11% /boot
/dev/sdd1        20G   14G  4.9G  75% /data
/dev/sdc1        20G  1.4G   18G   8% /opt
/dev/sdb1       2.0G  155M  1.8G   9% /var
/dev/sdf1       9.9G  2.4G  7.0G  26% /backup  

and everything is fine. i rebooted to test it. it happened again. only / and swap were mounted. blkid output:

   /dev/sdb1: UUID="3a11afe1-52d5-4e31-96a0-66da2c8e70eb" TYPE="ext3"
   /dev/sdf1: UUID="0e1f69a7-36d9-4af1-a537-afaa211e87d7" TYPE="ext3"
   /dev/sdb2: UUID="416970f8-c21b-419b-90d5-eb8eabb685a6" TYPE="swap"
   /dev/sdc1: UUID="d380ddf8-3476-46b3-8e80-9dd3b394dd13" TYPE="ext3"
   /dev/sde1: UUID="b224fa8a-e909-432e-927e-4a98fe2d74d0" TYPE="ext3"
   /dev/sda1: UUID="9407e385-168c-4e37-9651-1de04406b620" SEC_TYPE="ext2" 
   TYPE="ext3"
   /dev/sdd1: UUID="e8439366-d29d-43c3-ad5e-635855f4e42e" TYPE="ext3"

part of cat /etc/fstab output:

  UUID=9407e385-168c-4e37-9651-1de04406b620 /         ext3    defaults   0 0
  UUID=b224fa8a-e909-432e-927e-4a98fe2d74d0 /boot     ext3    defaults   0 0
  UUID=e8439366-d29d-43c3-ad5e-635855f4e42e /data     ext3    defaults   0 0
  UUID=d380ddf8-3476-46b3-8e80-9dd3b394dd13 /opt      ext3    defaults   0 0
  UUID=3a11afe1-52d5-4e31-96a0-66da2c8e70eb /var      ext3    defaults   0 0
  UUID=416970f8-c21b-419b-90d5-eb8eabb685a6 swap      swap    defaults   0 0
  UUID=0e1f69a7-36d9-4af1-a537-afaa211e87d7 /backup   ext3    defaults   0 0

why this happens?

  • Is the line split for the first line of fstab intentional? Or is that just a formatting error? – Haxiel Feb 18 at 7:49
  • lsblk doesn't tell you what's mounted. That's what mount is for. – roaima Feb 18 at 7:52
  • @Haxiel error.updated. – BlackCrystal Feb 18 at 10:49
4

Maybe someone or something has caused /etc/mtab to become un-writeable (immutable file? filesystem errors keeping the root filesystem read-only?), and it contains old data from before the reboot. As a result, the mount and df commands think the filesystems are already mounted, even though they really aren't.

Check for filesystems in read-only state: grep ro, /proc/mounts

Check for immutable /etc/mtab: lsattr /etc/mtab

Use the chattr -i /etc/mtab command to remove the immutable flag from /etc/mtab if necessary.

In modern systems, /etc/mtab is increasingly often a symbolic link to /proc/mounts or /proc/self/mounts. If your system/distribution has an old version of the mount command, such linking may cause the mount option user to fail, as it cannot record the name of the user that mounted a particular filesystem to /etc/mtab. If you don't use the user mount option, you can create this symbolic link on older systems too.

The advantage of this linking is that errors like you're experiencing cannot happen, as /proc/mounts (or /proc/self/mounts on systems with namespace support) always has up-to-date information on mounted filesystems, straight from the kernel itself.

Newer versions of the mount command will use /run/mount for this purpose instead if /etc/mtab is linked to /proc/self/mounts.

If you find /etc/mtab has been set immutable, and find no other reason for it, you might have been hacked, as making /etc/mtab immutable could be a way for an intruder to hide their tools from casual observation...

1

It's possible that your root filesystem was set read-only before the system shut down, and the umount commands used to unmount all disks were unable to remove entries from /etc/mtab. On boot, the mount -a command ("mount everything that's not already mounted") looked at /etc/mtab and determined everything was already mounted so no action was required.

This would explain all of your symptoms: - mounts failing to be applied at boot - df -h command showing essentially the same disk for several mountpoints - "not mounted" errors when you tried to unmount apparently mounted filesystems - mounting working after the umount attempts.

As has been suggested by telcoM it may be better to replace the file /etc/mtab with a symbolic link to /proc/mounts. But take this advice with caution: it may break your system if it's not set up to expect this.

0

i have read roaima and telcoM answers and found out that all filesystems including / are READ-ONLY. i checked kernel messages dmesg and found out that SELinux was preventing VM partitions to be mounted. well everthing was read only and i couldn't disable SELinux in /etc/selinux/config so i used UBUNTUlive, disabled SELinux, rebooted my system and everything is good and back to normal.

  • Good to hear it's sorted now. But please don't write an answer that says the same as one of the existing answers. Instead, use the tick-mark to accept the answer that best suited you. – roaima Feb 18 at 12:31
  • 1
    stopdisablingselinux.com – Fox Feb 23 at 8:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.