1

I have a Centos 7 vm that won't boot, the journalctl says

*failed to mount /sysroot. Dependency failed for Initrd Root File System. Dependency failed for Reload Configuration from the Real Root *

And instead of trying to just plain fix it, I'm opting to just mount the lvm inside and recover the data that way. The problem is everytime I try, it keeps saying

"mount /dev/mapper/centos-root on /mnt failed: No space left on device"

The thing is there should be enough space. The way I setup the recovery is, there are 3 hard drives in my server. One 250GB hdd as 'storage1' (Installed the esxi and mounted the original vm here), and two 4TB hdd's as 'storage2' (Made the new vm here that will mount the old vm). The old VM is around 3.5TB in size, and the new VM is 4.5TB.

Any help is appreciated.

--edit--

dmesg:

[ 2030.649327] XFS (dm-7): Mounting V4 Filesystem

[ 2030.759674] XFS (dm-7): Starting recovery (logdev: internal)

[ 2031.343184] XFS (dm-7): metadata I/O error: block 0x2416f8 ("xlog_recover_iodone") error 28 numblks 8

[ 2031.343194] XFS (dm-7): xfs_do_force_shutdown(0x1) called from line 378 of file fs/xfs/xfs_log_recover.c. Return address = 0xffffffffa023b9a5

[ 2031.343197] XFS (dm-7): I/O Error Detected. Shutting down filesystem

[ 2031.343199] XFS (dm-7): Please umount the filesystem and rectify the problem(s)

[ 2031.392868] XFS (dm-7): log mount/recovery failed: error -28

[ 2031.393489] XFS (dm-7): log mount failed

journalctl -k:

Feb 24 09:57:37 localhost.localdomain kernel: XFS (dm-7): Mounting V4 Filesystem

Feb 24 09:57:37 localhost.localdomain kernel: XFS (dm-7): Starting recovery (logdev: internal)

Feb 24 09:57:38 localhost.localdomain kernel: XFS (dm-7): metadata I/O error: block 0x2416f8 ("xlog_recover_iodone") error 28 numbl

Feb 24 09:57:38 localhost.localdomain kernel: XFS (dm-7): xfs_do_force_shutdown(0x1) called from line 378 of file fs/xfs/xfs_log_re

Feb 24 09:57:38 localhost.localdomain kernel: XFS (dm-7): I/O Error Detected. Shutting down filesystem

Feb 24 09:57:38 localhost.localdomain kernel: XFS (dm-7): Please umount the filesystem and rectify the problem(s)

Feb 24 09:57:38 localhost.localdomain kernel: XFS (dm-7): log mount/recovery failed: error -28

Feb 24 09:57:38 localhost.localdomain kernel: XFS (dm-7): log mount failed

fdisk /dev/sdb (the hard disk from the original VM)

Disk /dev/sdb: 3848.3 GB, 3848290697216 bytes, 7516192768 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk label type: dos

Disk identifier: 0x0009f290

Device Boot Start End Blocks Id System

/dev/sdb1 * 2048 1026047 512000 83 Linux

/dev/sdb2 1026048 524287999 261630976 8e Linux LVM

/dev/sdb3 524288000 4294967294 1885339647+ 8e Linux LVM

fdisk -l dev/centos/root (the physical volume I'm trying to mount since it's an LVM and I can't directly mount /dev/sdb)

Disk /dev/centos/root: 4040.5 GB, 4040490483712 bytes, 7891582976 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 131072 bytes / 131072 bytes

df -h shows /dev/mapper/test-root only has Use% at 1% so I really don't understand why it keeps saying No space left on device

  • What type of filesystem is on centos-root? (ext4, btrfs, XFS, etc.) Also, do you get any new dmesg/journalctl -k lines from that mount failing? – derobert Feb 24 '18 at 1:12
  • Thank you for the responce, the centos-root is an xfs. I'll update the post with what dmesg and journcalctl -k says – Joey Ponce Feb 24 '18 at 2:05
  • 1
    That sounds like you have a bad block, or some other storage failure — hopefully someone who knows XFS better than I can answer how to recover. I would guess xfs_repair, but I have no actual recent XFS experience. – derobert Feb 24 '18 at 9:26
  • running xfs_repair showed me that there was a bad superblock. But since trying to fix it seems more difficult, I've been trying to just mount it. I can successfully mount /dev/centos/home and I can see some files I left on the desktop so I'm fairly certain if I can just mount /dev/centos/root, I'll be able to recover the data. But I don't understand why it keeps saying that the device has no space left. I've tried mounting it so many times with different sized VM's and it always gives the same error. – Joey Ponce Feb 26 '18 at 9:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.