1

I have a production ESXI server with loads of VMs. I had a power outage a few hours ago that was so long my UPS's battery drained. The automatic shutdown mechanism wasn't working for some reason so the power was cut off for the whole system.

After the outage everything came up, except the mysql server VM. Now it spams the console with I/O errors.

end_request: critical medium error, dev sda, sector X
end_request: I/O error, dev sda, sector X
....
EXT4-fs error (device dm-1): ext4_wait_block_bitmap:476 comm bounce: Cannot read block bitmap - block_group = X, block_bitmap = X
Aborting journal on device dm-1-8
EXT4-fs (dm-1): Remounting filesystem read-only

Console

The VM is setup using encrypted LVM.

What do these errors mean? Is it hardware? What can I do? I searched Google for hours, but can't figure out what to do. I booted from live CD, run fsck on the unmounted root partition, fixed it, rebooted, but the issue is the same.

EDIT #1 I tried this, but nothing happened.

root@ubuntu:~# sudo cryptsetup --key-file=/media/ubuntu/7b225e2d-9c0f-4bd4-a4de-1d2f7a0b4c58/keyfile luksOpen /dev/sda5 myvolume

root@ubuntu:~# vgscan
  Reading all physical volumes.  This may take a while...
  Found volume group "mysql-server-vg" using metadata type lvm2

root@ubuntu:~# tune2fs -O ^has_journal /dev/mysql-server-vg/root 
tune2fs 1.42.13 (17-May-2015)
The needs_recovery flag is set.  Please run e2fsck before clearing
the has_journal flag.

root@ubuntu:~# e2fsck -f /dev/mysql-server-vg/root
\e2fsck 1.42.13 (17-May-2015)
/dev/mysql-server-vg/root: recovering journal
Pass 1: Checking inodes, blocks, and sizes
Deleted inode 391687 has zero dtime.  Fix? yes
Inodes that were part of a corrupted orphan linked list found.  Fix? yes
Inode 391697 was part of the orphaned inode list.  FIXED.
Inode 391699 was part of the orphaned inode list.  FIXED.
Inode 391700 was part of the orphaned inode list.  FIXED.
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Free blocks count wrong (5462594, counted=5462792).
Fix? yes
Inode bitmap differences:  -391687 -391697 -(391699--391700)
Fix? yes
Free inodes count wrong for group #48 (7946, counted=7950).
Fix? yes
Free inodes count wrong (1854371, counted=1854370).
Fix? yes

/dev/mysql-server-vg/root: ***** FILE SYSTEM WAS MODIFIED *****
/dev/mysql-server-vg/root: 95870/1950240 files (0.8% non-contiguous), 2337016/7799808 blocks

root@ubuntu:~# tune2fs -O ^has_journal /dev/mysql-server-vg/root 
tune2fs 1.42.13 (17-May-2015)

root@ubuntu:~# e2fsck -f /dev/mysql-server-vg/root
e2fsck 1.42.13 (17-May-2015)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/mysql-server-vg/root: 95870/1950240 files (0.8% non-contiguous), 2304248/7799808 blocks

root@ubuntu:~# tune2fs -j /dev/mysql-server-vg/root
tune2fs 1.42.13 (17-May-2015)
Creating journal inode: done

closed as off-topic by Jeff Schaller, Stephen Rauch, Satō Katsura, Michael Homer, GAD3R Sep 2 '17 at 16:19

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Questions describing a problem that can't be reproduced and seemingly went away on its own (or went away when a typo was fixed) are off-topic as they are unlikely to help future readers." – Jeff Schaller, Stephen Rauch, Satō Katsura, Michael Homer, GAD3R
If this question can be reworded to fit the rules in the help center, please edit the question.

2

OK, I figured it out and successfully fixed it. It cost me two days.

First I verified that the storage controller, the datastore hardware (mechanical drive) and the cables are not faulty. Please note that I couldn't access the vmdk file on the filesystem properly. I tried to copy it locally, with scp and with the vSphere Client, but after a while all of them gave me Input/Output error.

I even tried to clone the virtual disk to a separate datastore.

cd /vmfs/volumes/
vmkfstools -i datastore1/vm/vm.vmdk datastore2/vm/vm.vmdk -d thin -a lsilogic

It gave me Input/Output error after 16%.

I figured the power outage caused some corruption, stale locks and whatnots on the vmfs filesystem (datastore). Using the vSphere On-disk Metadata Analyzer (VOMA) I checked the VMFS metadata consistency. Please note that the datastore have to be unmounted before running this command.

voma -m vmfs -f check /vmfs/devices/disks/disk_name:1

It found 34 errors. The voma bundled in vSphere Hypervisor version 5.5 can only check the filesystem. I cloned the datastore to a new hard drive with clonezilla in rescue mode (cloning disk with bad sectors). After that I upgraded to VMware ESXi version 6.5, because it has a newer version of the voma command. It can fix errors, so I ran the following command:

voma -m vmfs -f fix /vmfs/devices/disks/disk_name:1

It sure did something. Booted up the VM, but cannot get console connection because of the new vCenter vSphere WebClient nonsense and vSphere Client deprecation, so I went back to my original VMware ESXi 5.5 installation. I cloned the mentioned vmdk file successfully. I booted up the VM with the cloned disk, ran fsck once, rebooted and voila. It works like expected. The server came online with all of my data.

It involved a lot of fiddling around, but I cannot figure out anything else. If somebody knows an easier way, please don't hesitate to leave a comment.

I did have database backup taken 12 hours before the incident, but wanted to recover the live data if possible.

Not the answer you're looking for? Browse other questions tagged or ask your own question.