mount: warning: /etc/mtab is not writable (e.g. read-only filesystem)

Question

Working on my system I noted that the root filesystem is mounted read-only. My system worked for months, today I get this error:

mount: warning: /etc/mtab is not writable (e.g. read-only filesystem).

Can you suggesting what to do?

I wonder why the root partition was mounted read-only and how was that possible.

$ mount
/dev/mapper/sysvg-lvroot on / type ext4 (rw,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/cgroup type tmpfs (rw)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
none on /run/user type tmpfs (rw,noexec,nosuid,nodev,size=104857600,mode=0755)
none on /sys/fs/pstore type pstore (rw)
/dev/mapper/sysvg-lvboot on /boot type ext4 (rw)
/dev/mapper/sysvg-lvvar on /var type ext4 (rw)
/dev/mapper/sysvg-lvusr on /usr type ext4 (rw)
/dev/mapper/sysvg-lvhome on /home type ext4 (rw)
rpc_pipefs on /run/rpc_pipefs type rpc_pipefs (rw)
systemd on /sys/fs/cgroup/systemd type cgroup (rw,noexec,nosuid,nodev,none,name=systemd)
gvfsd-fuse on /run/user/1000/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,user=paolo)
/dev/sdb1 on /media/paolo/SYSTEM type fuseblk (rw,nosuid,nodev,allow_other,default_permissions,blksize=4096)
/dev/sdb2 on /media/paolo/863E10053E0FECCD type fuseblk (rw,nosuid,nodev,allow_other,default_permissions,blksize=4096)
/dev/sdb8 on /media/paolo/905d8628-93c9-44d3-9619-02693334cc03 type ext4 (rw,nosuid,nodev,uhelper=udisks2)
/dev/sdb7 on /media/paolo/40be2a22-6db1-4fc5-a60c-c765641c0c52 type ext4 (rw,nosuid,nodev,uhelper=udisks2)
/dev/sdb6 on /media/paolo/df64c070-9916-4771-8bca-95e6214cda72 type ext3 (rw,nosuid,nodev,uhelper=udisks2)

mount: warning: /etc/mtab is not writable (e.g. read-only filesystem).
       It's possible that information reported by mount(8) is not
       up to date. For actual information about system mount points
       check the /proc/mounts file.


$ cat /proc/mount
rootfs / rootfs rw 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
udev /dev devtmpfs rw,relatime,size=4050100k,nr_inodes=1012525,mode=755 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=812256k,mode=755 0 0
/dev/mapper/sysvg-lvroot / ext4 ro,relatime,errors=remount-ro,data=ordered 0 0
none /sys/fs/cgroup tmpfs rw,relatime,size=4k,mode=755 0 0
none /sys/fs/fuse/connections fusectl rw,relatime 0 0
none /sys/kernel/debug debugfs rw,relatime 0 0
none /sys/kernel/security securityfs rw,relatime 0 0
none /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
none /run/shm tmpfs rw,nosuid,nodev,relatime 0 0
none /run/user tmpfs rw,nosuid,nodev,noexec,relatime,size=102400k,mode=755 0 0
none /sys/fs/pstore pstore rw,relatime 0 0
/dev/mapper/sysvg-lvboot /boot ext4 rw,relatime,data=ordered 0 0
/dev/mapper/sysvg-lvvar /var ext4 rw,relatime,data=ordered 0 0
/dev/mapper/sysvg-lvusr /usr ext4 rw,relatime,data=ordered 0 0
/dev/mapper/sysvg-lvhome /home ext4 rw,relatime,data=ordered 0 0
rpc_pipefs /run/rpc_pipefs rpc_pipefs rw,relatime 0 0
systemd /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,name=systemd 0 0
gvfsd-fuse /run/user/1000/gvfs fuse.gvfsd-fuse rw,nosuid,nodev,relatime,user_id=1000,group_id=1000 0 0
/dev/sdb1 /media/paolo/SYSTEM fuseblk rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other,blksize=4096 0 0
/dev/sdb2 /media/paolo/863E10053E0FECCD fuseblk rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other,blksize=4096 0 0
/dev/sdb8 /media/paolo/905d8628-93c9-44d3-9619-02693334cc03 ext4 rw,nosuid,nodev,relatime,data=ordered 0 0
/dev/sdb7 /media/paolo/40be2a22-6db1-4fc5-a60c-c765641c0c52 ext4 rw,nosuid,nodev,relatime,data=ordered 0 0
/dev/sdb6 /media/paolo/df64c070-9916-4771-8bca-95e6214cda72 ext3 rw,nosuid,nodev,relatime,data=ordered 0 0

This is how the smart status is reported by Disks (gnome-disk-utility).

Answer 1

Here's an excerpt from your kernel logs:

[194844.372691] ata1.00: exception Emask 0x0 SAct 0x700 SErr 0x0 action 0x0
[194844.372702] ata1.00: irq_stat 0x40000008
[194844.372710] ata1.00: failed command: READ FPDMA QUEUED
[194844.372723] ata1.00: cmd 60/08:40:98:cc:96/00:00:0b:00:00/40 tag 8 ncq 4096 in
[194844.372723]          res 41/40:00:98:cc:96/00:00:0b:00:00/40 Emask 0x409 (media error) <F>
[194844.372729] ata1.00: status: { DRDY ERR }
[194844.372734] ata1.00: error: { UNC }
[194844.384467] ata1.00: configured for UDMA/133
[194844.384495] sd 0:0:0:0: [sda] Unhandled sense code
[194844.384501] sd 0:0:0:0: [sda]  
[194844.384505] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[194844.384510] sd 0:0:0:0: [sda]  
[194844.384513] Sense Key : Medium Error [current] [descriptor]
[194844.384521] Descriptor sense data with sense descriptors (in hex):
[194844.384524]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
[194844.384542]         0b 96 cc 98
[194844.384550] sd 0:0:0:0: [sda]  
[194844.384555] Add. Sense: Unrecovered read error - auto reallocate failed
[194844.384560] sd 0:0:0:0: [sda] CDB:
[194844.384563] Read(10): 28 00 0b 96 cc 98 00 00 08 00
[194844.384586] end_request: I/O error, dev sda, sector 194432152
[194844.384599] ata1: EH complete
[194844.384628] EXT4-fs error (device dm-2): __ext4_get_inode_loc:3953: inode #32770: block 2721: comm cron: unable to read itable block
[194844.412579] Aborting journal on device dm-2-8.
[194844.412933] EXT4-fs (dm-2): Remounting filesystem read-only
[194844.412946] EXT4-fs error (device dm-2) in ext4_reserve_inode_write:4902: IO failure
[194844.514777] ------------[ cut here ]------------
[194844.514787] WARNING: CPU: 4 PID: 11053 at /build/buildd/linux-3.13.0/fs/ext4/ext4_jbd2.c:259 __ext4_handle_dirty_metadata+0x1a2/0x1c0()
[194844.514788] Modules linked in: usb_storage dm_snapshot pci_stub vboxpci(OX) vboxnetadp(OX) vboxnetflt(OX) vboxdrv(OX) bnep rfcomm bluetooth nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_core videodev snd_hda_codec_hdmi snd_hda_codec_idt lib80211_crypt_tkip hp_wmi sparse_keymap wl(POX) snd_hda_intel snd_hda_codec radeon hp_accel snd_hwdep intel_rapl snd_pcm i915 snd_page_alloc x86_pkg_temp_thermal snd_seq_midi intel_powerclamp snd_seq_midi_event ttm kvm_intel snd_rawmidi snd_seq kvm drm_kms_helper drm crct10dif_pclmul lib80211 lis3lv02d mei_me i2c_algo_bit snd_seq_device cfg80211 crc32_pclmul snd_timer mei input_polldev ghash_clmulni_intel snd aesni_intel rtsx_pci_ms aes_x86_64 lrw soundcore joydev gf128mul lpc_ich mac_hid memstick hp_wireless wmi glue_helper ablk_helper serio_raw video cryptd parport_pc ppdev coretemp lp parport hid_generic usbhid hid rtsx_pci_sdmmc ahci psmouse libahci sdhci_pci r8169 sdhci rtsx_pci mii
[194844.514843] CPU: 4 PID: 11053 Comm: cron Tainted: P           OX 3.13.0-43-generic #72-Ubuntu
[194844.514844] Hardware name: Hewlett-Packard HP Pavilion dv6 Notebook PC/17FA, BIOS F.1C 01/23/2013
[194844.514846]  0000000000000009 ffff88024ccf9970 ffffffff81720bf6 0000000000000000
[194844.514849]  ffff88024ccf99a8 ffffffff810677cd ffff88020a02f820 ffff880036509f70
[194844.514851]  ffff8802539194e0 ffffffff8182e3c0 000000000000017c ffff88024ccf99b8
[194844.514853] Call Trace:
[194844.514858]  [<ffffffff81720bf6>] dump_stack+0x45/0x56
[194844.514861]  [<ffffffff810677cd>] warn_slowpath_common+0x7d/0xa0
[194844.514863]  [<ffffffff810678aa>] warn_slowpath_null+0x1a/0x20
[194844.514866]  [<ffffffff8126e8b2>] __ext4_handle_dirty_metadata+0x1a2/0x1c0
[194844.514869]  [<ffffffff8124a492>] ext4_handle_dirty_dirent_node+0xc2/0x1b0
[194844.514872]  [<ffffffff81243844>] ? ext4_mark_inode_dirty+0x44/0x1f0
[194844.514874]  [<ffffffff8124be0e>] add_dirent_to_buf+0x1ee/0x230
[194844.514876]  [<ffffffff8124c70f>] ext4_add_entry+0x8bf/0xf30
[194844.514879]  [<ffffffff8124336c>] ? ext4_mark_iloc_dirty+0x31c/0x710
[194844.514881]  [<ffffffff8124cd9e>] ext4_add_nondir+0x1e/0x80
[194844.514892]  [<ffffffff8124d074>] ext4_create+0x104/0x170
[194844.514896]  [<ffffffff811c9bbd>] vfs_create+0xcd/0x130
[194844.514898]  [<ffffffff811cd23e>] do_last+0x103e/0x1230
[194844.514901]  [<ffffffff811ca801>] ? link_path_walk+0x71/0x870
[194844.514903]  [<ffffffff81314d7b>] ? apparmor_file_alloc_security+0x5b/0x180
[194844.514907]  [<ffffffff812d73c6>] ? security_file_alloc+0x16/0x20
[194844.514910]  [<ffffffff811cd4eb>] path_openat+0xbb/0x650
[194844.514912]  [<ffffffff811ce8ea>] do_filp_open+0x3a/0x90
[194844.514916]  [<ffffffff811db777>] ? __alloc_fd+0xa7/0x130
[194844.514919]  [<ffffffff811bcf69>] do_sys_open+0x129/0x280
[194844.514921]  [<ffffffff811bd0de>] SyS_open+0x1e/0x20
[194844.514924]  [<ffffffff817316ad>] system_call_fastpath+0x1a/0x1f
[194844.514926] ---[ end trace 863c5604899d0c53 ]---

The part starting with ata1.00: exception describes an error from the disk. This error affects the filesystem, and to preserve further damage, the kernel ends up “Remounting filesystem read-only”.

After that, there is a stack trace from the ext4 filesystem driver.

There are three possible explanations.

• It's a bug in the kernel — in the ext4 filesystem driver, or in the driver for your disk hardware, or in some other relevant part of the hardware.
• The disk is damaged, or the connection to it is bad (buggy controller chip, damaged or loose cable). This is the most obvious explanation for the drive hiccup that started it all. The filesystem driver may be crashing because it isn't fully robust against drive errors.
• Your RAM is damaged. This caused the kernel to misbehave in some way, perhaps sending an invalid command to the disk or misinterpreting its output.

The first thing to do is to test your RAM. RAM chips go bad over time. Run Memtest86+ for at least one full pass.

If the RAM passes, the disk is the next most likely culprit. Disks go bad over time too. Run smartctl to see if the disk reports any error.

If the hardware looks ok, check for any bug reports against your distribution's kernel. If you're running a kernel which isn't from a distribution, you may want to try using a more standard binary.

Answer 2

So the following line shows that you have a read-only file system at /...

/dev/mapper/sysvg-lvroot / ext4 ro,relatime,errors=remount-ro,data=ordered 0 0

But you should also have some indication that your file system has indeed have errors (e.g. from dmesg, on some systems also journalctl) and was mounted read-only because of that. Those errors should probably be fixed. You may also attempt to remount it as writable.

mount -o remount,rw /

There are many possible reasons for /etc/mtab not being writable and the most prominent ones are that distributions are switching to using a symlink /etc/mtab pointing into /proc.

In my case:

$ ls -l /etc/mtab
lrwxrwxrwx 1 root root 19 Aug  8 13:04 /etc/mtab -> ../proc/self/mounts

In other words, they are abandoning the concept of a writable mtab where mount stores its information. You may just be using a tool (e.g. a version of mount) that is not yet ready for such setup.