Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Short version: ext3 root filesystem on rackspace (xen) VM detects aborted journal on boot and mounts read-only. I've attempted to repair this from a rescue environment with tune2fs and e2fsck as prescribed in many articles I read, but the error continues to happen.

UPDATE: So based on this article I added "barrier=0" to the /etc/fstab entry for this filesystem and it mounted r/w fine at the next boot. I'm lead to believe this is a paravirtualization thing, but would love it if anyone fully understands what is going on here and can explain.

Long version:

Rackspace VM just upgraded from Ubuntu 11.10 to 12.04.2

dmesg output with the error:

[   14.701446] blkfront: barrier: empty write xvda op failed
[   14.701452] blkfront: xvda: barrier or flush: disabled
[   14.701460] end_request: I/O error, dev xvda, sector 28175816
[   14.701473] end_request: I/O error, dev xvda, sector 28175816
[   14.701487] Aborting journal on device xvda1.
[   14.704186] EXT3-fs (xvda1): error: ext3_journal_start_sb: Detected aborted journal
[   14.704199] EXT3-fs (xvda1): error: remounting filesystem read-only
[   14.940734] init: dmesg main process (763) terminated with status 7
[   18.425994] init: mongodb main process (769) terminated with status 1
[   21.940032] eth1: no IPv6 routers present
[   23.612044] eth0: no IPv6 routers present
[   27.147759] [UFW BLOCK] IN=eth0 OUT= MAC=40:40:73:00:ea:12:c4:71:fe:f1:e1:3f:08:00 SRC=98.143.36.192 DST=50.56.240.11 LEN=40 TOS=0x00 PREC=0x00 TTL=242 ID=37934 DF PROTO=TCP SPT=30269 DPT=8123 WINDOW=512 RES=0x00 SYN URGP=0 
[   31.025920] [UFW BLOCK] IN=eth0 OUT= MAC=40:40:73:00:ea:12:c4:71:fe:f1:e1:3f:08:00 SRC=116.6.60.9 DST=50.56.240.11 LEN=40 TOS=0x00 PREC=0x00 TTL=101 ID=256 PROTO=TCP SPT=6000 DPT=1433 WINDOW=16384 RES=0x00 SYN URGP=0 
[  493.974612] EXT3-fs (xvda1): error: ext3_remount: Abort forced by user
[  505.887555] EXT3-fs (xvda1): error: ext3_remount: Abort forced by user

In a rescue OS, I've tried:

tune2sf -O ^has_journal /dev/xdbb1 #Device is xvdb1 in rescue, but xvdba1 in real OS
e2fsck -f /dev/xvdb1
tune2sf -j /dev/xvdb1

I've also run e2fsck -p, e2fsck -f, and tune2fs -e continue. Here's the output of tune2fs -l.

tune2fs 1.41.14 (22-Dec-2010)
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          68910771-4026-4588-a62a-54eb992f4c6e
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype sparse_super large_file
Filesystem flags:         signed_directory_hash 
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              1245184
Block count:              4980480
Reserved block count:     199219
Free blocks:              2550830
Free inodes:              1025001
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      606
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
Filesystem created:       Thu Oct 20 21:34:53 2011
Last mount time:          Mon Apr  8 23:01:13 2013
Last write time:          Mon Apr  8 23:08:09 2013
Mount count:              0
Maximum mount count:      29
Last checked:             Mon Apr  8 23:04:49 2013
Check interval:           15552000 (6 months)
Next check after:         Sat Oct  5 23:04:49 2013
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:           256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      1e07317a-6301-41d9-8885-0e3e837f2a38
Journal backup:           inode blocks

I also grepped some lines from /var/log/syslog while in the rescue mode with some additional error info:

Apr  8 19:47:06 dev kernel: [26504959.895754] blkfront: barrier: empty write xvda op failed
Apr  8 19:47:06 dev kernel: [26504959.895763] blkfront: xvda: barrier or flush: disabled
Apr  8 20:19:33 dev kernel: [    0.000000] Command line: root=/dev/xvda1 console=hvc0 ro quiet splash 
Apr  8 20:19:33 dev kernel: [    0.000000] Kernel command line: root=/dev/xvda1 console=hvc0 ro quiet splash 
Apr  8 20:19:33 dev kernel: [    0.240303] blkfront: xvda: barrier: enabled
Apr  8 20:19:33 dev kernel: [    0.249960]  xvda: xvda1
Apr  8 20:19:33 dev kernel: [    0.250356] xvda: detected capacity change from 0 to 20401094656
Apr  8 20:19:33 dev kernel: [    5.684101] EXT3-fs (xvda1): mounted filesystem with ordered data mode
Apr  8 20:19:33 dev kernel: [  140.547468] blkfront: barrier: empty write xvda op failed
Apr  8 20:19:33 dev kernel: [  140.547477] blkfront: xvda: barrier or flush: disabled
Apr  8 20:19:33 dev kernel: [  140.709985] EXT3-fs (xvda1): using internal journal
Apr  8 21:18:12 dev kernel: [    0.000000] Command line: root=/dev/xvda1 console=hvc0 ro quiet splash 
Apr  8 21:18:12 dev kernel: [    0.000000] Kernel command line: root=/dev/xvda1 console=hvc0 ro quiet splash 
Apr  8 21:18:12 dev kernel: [    1.439023] blkfront: xvda: barrier: enabled
Apr  8 21:18:12 dev kernel: [    1.454307]  xvda: xvda1
Apr  8 21:18:12 dev kernel: [    6.799014] EXT3-fs (xvda1): recovery required on readonly filesystem
Apr  8 21:18:12 dev kernel: [    6.799020] EXT3-fs (xvda1): write access will be enabled during recovery
Apr  8 21:18:12 dev kernel: [    6.839498] blkfront: barrier: empty write xvda op failed
Apr  8 21:18:12 dev kernel: [    6.839505] blkfront: xvda: barrier or flush: disabled
Apr  8 21:18:12 dev kernel: [    6.854814] EXT3-fs (xvda1): warning: ext3_clear_journal_err: Filesystem error recorded from previous mount: IO failure
Apr  8 21:18:12 dev kernel: [    6.854820] EXT3-fs (xvda1): warning: ext3_clear_journal_err: Marking fs in need of filesystem check.
Apr  8 21:18:12 dev kernel: [    6.855247] EXT3-fs (xvda1): recovery complete
Apr  8 21:18:12 dev kernel: [    6.855902] EXT3-fs (xvda1): mounted filesystem with ordered data mode
Apr  8 21:18:12 dev kernel: [  143.505890] EXT3-fs (xvda1): using internal journal
share|improve this question
1  
I/O error, dev xvda, sector 28175816, bad disk? –  frostschutz Apr 9 '13 at 0:26
    
I don't think so. It's a VM for one thing. I suspect this has something to do with the linux-generic kernel in combination with xen paravirtualization. I think the root cause maybe comes down to this bug: lzone.de/blkfront+barrier+empty+write+xvda+op+failed but not certain of that. –  Peter Lyons Apr 9 '13 at 3:31
    
A VM doesn't repair outer I/O errors, of course. I am not familiar with Xen, though. If this VM does not access the disk directly then have a look at the logs of the kernel with direct hardware access. Doesn't make sense to me that this I/O error shall be related to barriers (disabled barriers!). You may try to read the sector directly with dd to check for hardware errors. If you like fun of the "VERY DANGEROUS" category you may try to fix this with hdparm --write-sector from the kernel with direct hardware access. –  Hauke Laging Apr 9 '13 at 4:02
add comment

2 Answers

At this point I'm thinking this is very likely an instance of Debian Bug 637234. As this is a cloud VM, the hypervisor kernel is outside of my control. The workaround is using barrier=0 in /etc/fstab for the root filesystem. The long-term fix is to rebuild the box as a next-gen rackspace cloud instance instead of a first-gen Xen-based instance.

share|improve this answer
add comment

"barrier=0" in /etc/fstab is propably too late (and just comes into play after the filesystems get mounted RW in a later boot stage).

"barrier=off" as additional kernel-Parameter should work earlier and better.

Try it. If your DomU is being started by pygrub from the Dom0 (which is the "usual" way) you can put that into your grub-kernel-konfiguration-line in your DomU.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.