11

I was trying to install Ruby on my remote server. (This is a virtual machine (Debian) in a ESXi server.) I got this error:

$ sudo apt-get install ruby1.8
W: Not using locking for read only lock file /var/lib/dpkg/lock
E: Unable to write to /var/cache/apt/
E: The package lists or status file could not be parsed or opened.

Then I tried :

$ sudo dpkg --configure -a 
dpkg: unable to access dpkg status area: Read-only file system 
$ mount
/dev/sda3 on / type ext4 (rw,errors=remount-ro)
tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
udev on /dev type tmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620)
/dev/sda1 on /boot type ext4 (rw)
/dev/sdb1 on /home type ext4 (rw)
    
mount: warning: /etc/mtab is not writable (e.g. read-only filesystem).
       It's possible that information reported by mount(8) is not
       up to date. For actual information about system mount points
       check the /proc/mounts file.
$ cat /proc/mounts
rootfs / rootfs rw 0 0
none /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
none /proc proc rw,nosuid,nodev,noexec,relatime 0 0
none /dev devtmpfs rw,relatime,size=1553128k,nr_inodes=216450,mode=755 0 0
none /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
/dev/disk/by-uuid/cf4fb4ae-6d12-407b-bf43-3b0daaaaaf74 / ext4 ro,relatime,errors=remount-ro,barrier=1,data=ordered 0 0
tmpfs /lib/init/rw tmpfs rw,nosuid,relatime,mode=755 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev,relatime 0 0
/dev/sda1 /boot ext4 rw,relatime,barrier=1,data=ordered 0 0
/dev/sdb1 /home ext4 rw,relatime,barrier=1,data=ordered 0 0
# dmesg
…
[1968636.237601] JBD2: Detected IO errors while flushing file data on sdb1-8
[1968772.229102] JBD2: Detected IO errors while flushing file data on sdb1-8
[1968789.799409] IPv6 addrconf: prefix with wrong length 56
[1968990.325125] IPv6 addrconf: prefix with wrong length 56
[1969190.801848] IPv6 addrconf: prefix with wrong length 56
[1969192.245363] JBD2: Detected IO errors while flushing file data on sdb1-8
[1969197.698223] IPv6 addrconf: prefix with wrong length 56
[1969223.105506] JBD2: Detected IO errors while flushing file data on sdb1-8
[1969349.119764] JBD2: Detected IO errors while flushing file data on sdb1-8
[1969398.205686] IPv6 addrconf: prefix with wrong length 56
[1969598.713179] IPv6 addrconf: prefix with wrong length 56
[1969607.241633] JBD2: Detected IO errors while flushing file data on sdbΩ-8
[1969799.220758] IPv6 addrconf: prefix with wrong length 56
[1969825.462909] JBD2: Detected IO errors while flushing file data on sdb1-8
[1969831.231049] JBD2: Detected IO errors while flushing file data on sdb1-8
[1969999.728348] IPv6 addrconf: prefix with wrong length 56
[1970200.247944] IPv6 addrconf: prefix with wrong length 56
[1970221.321558] JBD2: Detected IO errors while flushing file data on sdb1-8
[1970253.105491] JBD2: Detected IO errors while flushing file data on sdb1-8
# nano /var/log/syslog
Aug 22 21:47:25 web01 kernel: [1333920.765599]  f6442200 00000082 f6a0c640 c10ce21a f7cc78bd c141d100 c141d100 c14186ac
Aug 22 21:47:25 web01 kernel: [1333920.765603]  f64423bc c3008100 00000000 0000000c ffffffff f8293c04 f6558000 004ac800
Aug 22 21:47:25 web01 kernel: [1333920.765606]  00000001 f64423bc 13dee727 00000663 c10ce260 00001000 de619d70 c2cd6f7c
Aug 22 21:47:25 web01 kernel: [1333920.765610] Call Trace:
Aug 22 21:47:25 web01 kernel: [1333920.765616]  [<c10ce21a>] ? __find_get_block+0x163/0x16d
Aug 22 21:47:25 web01 kernel: [1333920.765626]  [<f7cc78bd>] ? e1000_xmit_frame+0xa4d [e1000]
Aug 22 21:47:25 web01 kernel: [1333920.765629]  [<c10ce260>] ? __getblk+0x3c/0x2f3
Aug 22 21:47:25 web01 kernel: [1333920.765634]  [<f7eecfc4>] ? do_get_write_access+0x1f6/0x3d1 [jbd2]
Aug 22 21:47:25 web01 kernel: [1333920.765638]  [<c104435b>] ? wake_bit_function+0x0/0x3c
Aug 22 21:47:25 web01 kernel: [1333920.765642]  [<f7eed71d>] ? start_this_handle+0x2fb/0x37e [jbd2]
Aug 22 21:47:25 web01 kernel: [1333920.765646]  [<f7eed1b7>] ? jbd2_journal_get_write_access+0x18/0x26 [jbd2]
Aug 22 21:47:25 web01 kernel: [1333920.765655]  [<f7f8e355>] ? __ext4_journal_get_write_access+0x38/0x40 [ext4]
Aug 22 21:47:25 web01 kernel: [1333920.765662]  [<f7f718b5>] ? ext4_reserve_inode_write+0x27/0x57 [ext4]
Aug 22 21:47:25 web01 kernel: [1333920.765668]  [<f7f71914>] ? ext4_mark_inode_dirty+0x2f/0x1a9 [ext4]
Aug 22 21:47:25 web01 kernel: [1333920.765676]  [<f7f879fa>] ? ext4_journal_start_sb+0xb4/0xe3 [ext4]
Aug 22 21:47:25 web01 kernel: [1333920.765682]  [<f7f71b6a>] ? ext4_dirty_inode+0x24/0x35 [ext4]
Aug 22 21:47:25 web01 kernel: [1333920.765685]  [<c10c99ec>] ? __mark_inode_dirty+0x20/0x10b
Aug 22 21:47:25 web01 kernel: [1333920.765688]  [<c10c2363>] ? file_update_time+0xbd/0xde
Aug 22 21:47:25 web01 kernel: [1333920.765691]  [<c1088cf0>] ? __generic_file_aio_write+0x142/0x28d
Aug 22 21:47:25 web01 kernel: [1333920.765694]  [<c103b3a0>] ? lock_timer_base+0x19/0x35
Aug 22 21:47:25 web01 kernel: [1333920.765696]  [<c1088e8c>] ? generic_file_aio_write+0x51/0x93
Aug 22 21:47:25 web01 kernel: [1333920.765700]  [<c10b382a>] ? do_sync_write+0xc0/0x107
Aug 22 21:47:25 web01 kernel: [1333920.765702]  [<c104432e>] ? autoremove_wake_function+0x0/0x2d
Aug 22 21:47:25 web01 kernel: [1333920.765706]  [<c1102fe0>] ? security_file_permission+0xc/0xd
Aug 22 21:47:25 web01 kernel: [1333920.765708]  [<c10b376a>] ? do_sync_write+0x0/0x107
Aug 22 21:46:25 web01 kernel: [1333920.765710]  [<c10b4134>] ? vfs_write+0x7e/0xd6
Aug 22 21:47:25 web01 kernel: [1333920.765712]  [<c10b4224>] ? sys_write+0x3c/0x63
Aug 22 21:47:25 web01 kernel: [1333920.765715]  [<c10030fb>] ? sysenter_do_call+0x12/0x28
Aug 22 21:48:32 web01 kernel: [1333987.640773] sd 0:0:0:0: timing out command, waited 180s
Aug 22 21:48:32 web01 kernel: [1333987.640858] sd 0:0:0:0: [sda] Unhandled error code
Aug 22 21:48:32 web01 kernel: [1333987.640860] sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
Aug 22 21:48:32 web01 kernel: [1333987.640862] sd 0:0:0:0: [sda] CDB: Write(10): 2a 00 00 43 08 08 00 00 08 00
Aug 22 21:48:32 web01 kernel: [1333987.640867] end_request: I/O error, dev sda, sector 4392968
Aug 22 21:48:32 web01 kernel: [1333987.640928] Buffer I/O error on device sda3, logical block 1
Aug 22 21:48:32 web01 kernel: [1333987.640988] lost page write due to I/O error on sda3
Aug 22 21:48:32 web01 kernel: [1333987.641000] sd 0:0:0:0: timing out command, waited 180s
2
  • The output of mount is not reliable, especially since your root fs appears to be read-only. Can you post the output of cat /proc/mounts as well? – mrb Aug 29 '12 at 17:06
  • please check the update2 – Subhransu Mishra Aug 29 '12 at 17:11
12

Your root filesystem (/) is mounted read-only as the /dev/disk/... line in /proc/mounts shows. The reason for this could be that a disk error was detected on boot (errors=remount-ro option) or subsequent I/O error.

Check the kernel logs for any errors with the dmesg command and look in /var/log/syslog or /var/log/messages (but note that these files do probably not contain the last log entries). If there are any I/O errors, you may need to replace the disk. If not, boot to single user mode and run fsck.ext4 UUID=cf4fb4ae-6d12-407b-bf43-3b0daaaaaf7 to try to fix the errors.

If fsck reports no errors and it is still read-only on reboot, you can run:

sudo mount / -o remount,rw 

to try to mount the disk read-write.

Looking at the log files in /var/log/ does not help much, as it is at the moment read-only.


Often, the reason why /var/lib/dpkg/lock cannot be locked is that an automatic system update runs in the background, but in your case it is specifically complaining about a read-only filesystem.

6
  • I was going to +1 this but it's dangerous to just blindly remount rw unless you know why it's ro in the first place. It's better to just reboot into single user mode and run fsck. Only if it's clean (or fixed) should you boot back to rw state. If it's clean and still mounting ro, then perhaps try to force things. – bahamat Aug 29 '12 at 21:21
  • @bahamat Thanks for your comment, I changed my answer accordingly. – jofel Aug 29 '12 at 21:30
  • @jofel Please check the update3 - the output of dmesg , it seams like its io problem . what should be my next step ? – Subhransu Mishra Aug 30 '12 at 2:48
  • @jofel should I do a tune2fs -c 0 -i 0d /dev/sda3 and reboot the system ? – Subhransu Mishra Aug 30 '12 at 2:58
  • @Subhransu Depending how important the data on your root system partition is, I would try to make a backup first. I would then boot into single user mode (recovery mode) and then run fsck manually. The automatic fsck does often only report errors but does not fix them. So avoid unnecessary IO on a possible broken disk. The SMART-feature of the disk sometimes helps to detect if the disk is really broken. – jofel Aug 30 '12 at 13:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.