3

I am struggling getting a customer's server back up.Apparently the machine just slowed down and when they rebooted it, it booted into dracut claiming it can't find the UUID of the disks.

The original O/S that ran on the now missing LVM is RHEL 7. I added a 10Gb drive and installed CentOS 7 onto the new drive in order to have a live system to troubleshoot from.

'Naturally' there's no backups...

#pvs -a
PV         VG Fmt Attr PSize PFree
/dev/sda1         ---     0     0
/dev/sda2         ---     0     0
/dev/sdb1         ---     0     0
/dev/sdc1         ---     0     0
/dev/sdc2         ---     0     0
/dev/sdc3         ---     0     0

/dev/sdc is the new drive, I am trying to recover sda2 and sdb1.

# lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
fd0      2:0    1     4K  0 disk
sda      8:0    0   500G  0 disk
├─sda1   8:1    0   500M  0 part
└─sda2   8:2    0 499.5G  0 part
sdb      8:16   0     2T  0 disk
└─sdb1   8:17   0     2T  0 part
sdc      8:32   0    10G  0 disk
├─sdc1   8:33   0     1G  0 part /boot
├─sdc2   8:34   0     1G  0 part [SWAP]
└─sdc3   8:35   0     8G  0 part /
sr0     11:0    1  1024M  0 rom

PVScan reveals nothing

 # pvscan
  No matching physical volumes found

Fdisk still reports LVM partitions

# fdisk -l

Disk /dev/sda: 536.9 GB, 536870912000 bytes, 1048576000 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000a2e76

Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048     1026047      512000   83  Linux
/dev/sda2         1026048  1048575999   523774976   8e  Linux LVM

Disk /dev/sdc: 10.7 GB, 10737418240 bytes, 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000ddc56

Device Boot      Start         End      Blocks   Id  System
/dev/sdc1   *        2048     2099199     1048576   83  Linux
/dev/sdc2         2099200     4196351     1048576   82  Linux swap / Solaris
/dev/sdc3         4196352    20971519     8387584   83  Linux

Disk /dev/sdb: 2199.0 GB, 2199023255552 bytes, 4294967296 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xde7fc33e

Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048  4294967294  2147482623+  8e  Linux LVM

What really worries me, is that it seems the beginning of the disk is empty:

# dd if=/dev/sdb1 bs=512 count=255 skip=1
255+0 records in
255+0 records out
130560 bytes (131 kB) copied, 0.00671742 s, 19.4 MB/s

** EDIT: Update without cache ** VGScan output:

#lvmcmdline.c:2829          Parsing: vgscan -vvvv
#lvmcmdline.c:1881          Recognised command vgscan_general (id 141 / enum 127).
#config/config.c:1480        devices/global_filter not found in config: defaulting to global_filter = [ "a|.*/|" ]
#libdm-config.c:1002        global/lvmetad_update_wait_time not found in config: defaulting to 10
#daemon-client.c:33            /run/lvm/lvmetad.socket: Opening daemon socket to lvmetad for protocol lvmetad version 1.
#daemon-client.c:52            Sending daemon lvmetad: hello
#cache/lvmetad.c:143           Successfully connected to lvmetad on fd 3.
#filters/filter-sysfs.c:327           Sysfs filter initialised.
#filters/filter-internal.c:77            Internal filter initialised.
#filters/filter-type.c:56            LVM type filter initialised.
#filters/filter-usable.c:183           Usable device filter initialised.
#filters/filter-mpath.c:291           mpath filter initialised.
#filters/filter-partitioned.c:69            Partitioned filter initialised.
#filters/filter-md.c:169           MD filter initialised.
#filters/filter-composite.c:109           Composite filter initialised.
#config/config.c:1480        devices/filter not found in config: defaulting to filter = [ "a|.*/|" ]
#filters/filter-regex.c:216           Regex filter initialised.
#filters/filter-usable.c:183           Usable device filter initialised.
#filters/filter-composite.c:109           Composite filter initialised.
#libdm-config.c:975         devices/cache not found in config: defaulting to /etc/lvm/cache/.cache
#filters/filter-persistent.c:415           Persistent filter initialised.
#filters/filter-composite.c:109           Composite filter initialised.
#libdm-config.c:1074        metadata/record_lvs_history not found in config: defaulting to 0
#lvmcmdline.c:2897          DEGRADED MODE. Incomplete RAID LVs will be processed.
#lvmcmdline.c:2903          Processing command: vgscan -vvvv
#lvmcmdline.c:2904          Command pid: 10060
#lvmcmdline.c:2905          System ID:
#lvmcmdline.c:2908          O_DIRECT will be used
#locking/locking.c:129         File-based locking selected.
#libdm-common.c:984           Preparing SELinux context for /run/lock/lvm to system_u:object_r:lvm_lock_t:s0.
#libdm-common.c:987           Resetting SELinux context to default value.
#misc/lvm-flock.c:202         Locking /run/lock/lvm/P_global WB
#libdm-common.c:984           Preparing SELinux context for /run/lock/lvm/P_global to system_u:object_r:lvm_lock_t:s0.
#misc/lvm-flock.c:100           _do_flock /run/lock/lvm/P_global:aux WB
#misc/lvm-flock.c:100           _do_flock /run/lock/lvm/P_global WB
#misc/lvm-flock.c:47            _undo_flock /run/lock/lvm/P_global:aux
#libdm-common.c:987           Resetting SELinux context to default value.
#cache/lvmcache.c:2539          Dropping VG info
#cache/lvmcache.c:755           lvmcache has no info for vgname "#orphans_lvm1" with VGID #orphans_lvm1.
#cache/lvmcache.c:755           lvmcache has no info for vgname "#orphans_lvm1".
#cache/lvmcache.c:2086          lvmcache: Initialised VG #orphans_lvm1.
#cache/lvmcache.c:755           lvmcache has no info for vgname "#orphans_pool" with VGID #orphans_pool.
#cache/lvmcache.c:755           lvmcache has no info for vgname "#orphans_pool".
#cache/lvmcache.c:2086          lvmcache: Initialised VG #orphans_pool.
#cache/lvmcache.c:755           lvmcache has no info for vgname "#orphans_lvm2" with VGID #orphans_lvm2.
#cache/lvmcache.c:755           lvmcache has no info for vgname "#orphans_lvm2".
#cache/lvmcache.c:2086          lvmcache: Initialised VG #orphans_lvm2.
#cache/lvmetad.c:256           Sending lvmetad get_global_info
#label/label.c:833           Finding devices to scan
#device/dev-cache.c:1212          Creating list of system devices.
#device/dev-cache.c:723           Found dev 11:0 /dev/sr0 - new.
#device/dev-cache.c:763           Found dev 11:0 /dev/cdrom - new alias.
#device/dev-cache.c:763           Found dev 11:0 /dev/disk/by-id/ata-VMware_Virtual_SATA_CDRW_Drive_00000000000000000001 - new alias.
#device/dev-cache.c:763           Found dev 11:0 /dev/disk/by-path/pci-0000:02:01.0-ata-1.0 - new alias.
#device/dev-cache.c:723           Found dev 8:0 /dev/sda - new.
#device/dev-cache.c:763           Found dev 8:0 /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:0:0 - new alias.
#device/dev-cache.c:723           Found dev 8:1 /dev/sda1 - new.
#device/dev-cache.c:763           Found dev 8:1 /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:0:0-part1 - new alias.
#device/dev-cache.c:763           Found dev 8:1 /dev/disk/by-uuid/530982ac-2bdd-4310-857b-ff132f4a4d41 - new alias.
#device/dev-cache.c:723           Found dev 8:2 /dev/sda2 - new.
#device/dev-cache.c:763           Found dev 8:2 /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:0:0-part2 - new alias.
#device/dev-cache.c:763           Found dev 8:2 /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:0:0-part2 - new alias.
#device/dev-cache.c:723           Found dev 8:16 /dev/sdb - new.
#device/dev-cache.c:763           Found dev 8:16 /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:1:0 - new alias.
#device/dev-cache.c:723           Found dev 8:17 /dev/sdb1 - new.
#device/dev-cache.c:763           Found dev 8:17 /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:1:0-part1 - new alias.
#device/dev-cache.c:723           Found dev 8:32 /dev/sdc - new.
#device/dev-cache.c:763           Found dev 8:32 /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:2:0 - new alias.
#device/dev-cache.c:723           Found dev 8:33 /dev/sdc1 - new.
#device/dev-cache.c:763           Found dev 8:33 /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:2:0-part1 - new alias.
#device/dev-cache.c:763           Found dev 8:33 /dev/disk/by-uuid/0a81d8a5-6edd-40fb-ae16-14684f045ef7 - new alias.
#device/dev-cache.c:723           Found dev 8:34 /dev/sdc2 - new.
#device/dev-cache.c:763           Found dev 8:34 /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:2:0-part2 - new alias.
#device/dev-cache.c:763           Found dev 8:34 /dev/disk/by-uuid/52eb5eff-aa0c-435e-809c-5fbda45c6403 - new alias.
#device/dev-cache.c:723           Found dev 8:35 /dev/sdc3 - new.
#device/dev-cache.c:763           Found dev 8:35 /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:2:0-part3 - new alias.
#device/dev-cache.c:763           Found dev 8:35 /dev/disk/by-uuid/0dba35bd-de0b-4e59-b579-f9d3b18024d4 - new alias.
#device/dev-cache.c:723           Found dev 2:0 /dev/fd0 - new.
#filters/filter-type.c:27            /dev/fd0: Skipping: Unrecognised LVM device type 2
#device/dev-io.c:609           Opened /dev/sda RO O_DIRECT
#device/dev-io.c:359         /dev/sda: size is 1048576000 sectors
#device/dev-io.c:658           Closed /dev/sda
#filters/filter-partitioned.c:30            filter partitioned deferred /dev/sda
#filters/filter-md.c:99            filter md deferred /dev/sda
#filters/filter-persistent.c:346           filter caching good /dev/sda
#filters/filter-type.c:27            /dev/cdrom: Skipping: Unrecognised LVM device type 11
#device/dev-io.c:609           Opened /dev/sda1 RO O_DIRECT
#device/dev-io.c:359         /dev/sda1: size is 1024000 sectors
#device/dev-io.c:658           Closed /dev/sda1
#filters/filter-mpath.c:196           /dev/sda1: Device is a partition, using primary device sda for mpath component detection
#filters/filter-partitioned.c:30            filter partitioned deferred /dev/sda1
#filters/filter-md.c:99            filter md deferred /dev/sda1
#filters/filter-persistent.c:346           filter caching good /dev/sda1
#device/dev-io.c:609           Opened /dev/sda2 RO O_DIRECT
#device/dev-io.c:359         /dev/sda2: size is 1047549952 sectors
#device/dev-io.c:658           Closed /dev/sda2
#filters/filter-mpath.c:196           /dev/sda2: Device is a partition, using primary device sda for mpath component detection
#filters/filter-partitioned.c:30            filter partitioned deferred /dev/sda2
#filters/filter-md.c:99            filter md deferred /dev/sda2
#filters/filter-persistent.c:346           filter caching good /dev/sda2
#device/dev-io.c:609           Opened /dev/sdb RO O_DIRECT
#device/dev-io.c:359         /dev/sdb: size is 4294967296 sectors
#device/dev-io.c:658           Closed /dev/sdb
#filters/filter-partitioned.c:30            filter partitioned deferred /dev/sdb
#filters/filter-md.c:99            filter md deferred /dev/sdb
#filters/filter-persistent.c:346           filter caching good /dev/sdb
#device/dev-io.c:609           Opened /dev/sdb1 RO O_DIRECT
#device/dev-io.c:359         /dev/sdb1: size is 4294965247 sectors
#device/dev-io.c:658           Closed /dev/sdb1
#filters/filter-mpath.c:196           /dev/sdb1: Device is a partition, using primary device sdb for mpath component detection
#filters/filter-partitioned.c:30            filter partitioned deferred /dev/sdb1
#filters/filter-md.c:99            filter md deferred /dev/sdb1
#filters/filter-persistent.c:346           filter caching good /dev/sdb1
#device/dev-io.c:609           Opened /dev/sdc RO O_DIRECT
#device/dev-io.c:359         /dev/sdc: size is 20971520 sectors
#device/dev-io.c:658           Closed /dev/sdc
#filters/filter-partitioned.c:30            filter partitioned deferred /dev/sdc
#filters/filter-md.c:99            filter md deferred /dev/sdc
#filters/filter-persistent.c:346           filter caching good /dev/sdc
#device/dev-io.c:609           Opened /dev/sdc1 RO O_DIRECT
#device/dev-io.c:359         /dev/sdc1: size is 2097152 sectors
#device/dev-io.c:658           Closed /dev/sdc1
#filters/filter-mpath.c:196           /dev/sdc1: Device is a partition, using primary device sdc for mpath component detection
#filters/filter-partitioned.c:30            filter partitioned deferred /dev/sdc1
#filters/filter-md.c:99            filter md deferred /dev/sdc1
#filters/filter-persistent.c:346           filter caching good /dev/sdc1
#device/dev-io.c:609           Opened /dev/sdc2 RO O_DIRECT
#device/dev-io.c:359         /dev/sdc2: size is 2097152 sectors
#device/dev-io.c:658           Closed /dev/sdc2
#filters/filter-mpath.c:196           /dev/sdc2: Device is a partition, using primary device sdc for mpath component detection
#filters/filter-partitioned.c:30            filter partitioned deferred /dev/sdc2
#filters/filter-md.c:99            filter md deferred /dev/sdc2
#filters/filter-persistent.c:346           filter caching good /dev/sdc2
#device/dev-io.c:609           Opened /dev/sdc3 RO O_DIRECT
#device/dev-io.c:359         /dev/sdc3: size is 16775168 sectors
#device/dev-io.c:658           Closed /dev/sdc3
#filters/filter-mpath.c:196           /dev/sdc3: Device is a partition, using primary device sdc for mpath component detection
#filters/filter-partitioned.c:30            filter partitioned deferred /dev/sdc3
#filters/filter-md.c:99            filter md deferred /dev/sdc3
#filters/filter-persistent.c:346           filter caching good /dev/sdc3
#label/label.c:869           Found 9 devices to scan
#label/label.c:628           Scanning 9 devices for VG info
#label/label.c:664           Scanning submitted 9 reads
#label/label.c:683           Processing data from device /dev/sda 8:0 fd 4 block 0x56207e2a8870
#label/label.c:407         /dev/sda: No lvm label detected
#label/label.c:412           <backtrace>
#label/label.c:683           Processing data from device /dev/sda1 8:1 fd 6 block 0x56207e2a88c0
#label/label.c:407         /dev/sda1: No lvm label detected
#label/label.c:412           <backtrace>
#label/label.c:683           Processing data from device /dev/sda2 8:2 fd 7 block 0x56207e2a8910
#label/label.c:407         /dev/sda2: No lvm label detected
#label/label.c:412           <backtrace>
#label/label.c:683           Processing data from device /dev/sdb 8:16 fd 8 block 0x56207e2a8960
#label/label.c:407         /dev/sdb: No lvm label detected
#label/label.c:412           <backtrace>
#label/label.c:683           Processing data from device /dev/sdb1 8:17 fd 9 block 0x56207e2a89b0
#label/label.c:407         /dev/sdb1: No lvm label detected
#label/label.c:412           <backtrace>
#label/label.c:683           Processing data from device /dev/sdc 8:32 fd 10 block 0x56207e2a8a00
#label/label.c:407         /dev/sdc: No lvm label detected
#label/label.c:412           <backtrace>
#label/label.c:683           Processing data from device /dev/sdc1 8:33 fd 11 block 0x56207e2a8a50
#label/label.c:407         /dev/sdc1: No lvm label detected
#label/label.c:412           <backtrace>
#label/label.c:683           Processing data from device /dev/sdc2 8:34 fd 12 block 0x56207e2a8aa0
#label/label.c:407         /dev/sdc2: No lvm label detected
#label/label.c:412           <backtrace>
#label/label.c:683           Processing data from device /dev/sdc3 8:35 fd 13 block 0x56207e2a8af0
#label/label.c:407         /dev/sdc3: No lvm label detected
#label/label.c:412           <backtrace>
#label/label.c:763           Scanned devices: read errors 0 process errors 0 failed 0
#cache/lvmetad.c:2374      Scanning all devices to update lvmetad.
#cache/lvmetad.c:558           Sending lvmetad token_update update in progress
#cache/lvmetad.c:2412          Telling lvmetad to clear its cache
#filters/filter-type.c:27            /dev/fd0: Skipping: Unrecognised LVM device type 2
#device/dev-io.c:336         /dev/sda: using cached size 1048576000 sectors
#filters/filter-partitioned.c:37            /dev/sda: Skipping: Partition table signature found
#filters/filter-type.c:27            /dev/cdrom: Skipping: Unrecognised LVM device type 11
#device/dev-io.c:336         /dev/sda1: using cached size 1024000 sectors
#filters/filter-mpath.c:196           /dev/sda1: Device is a partition, using primary device sda for mpath component detection
#device/dev-io.c:336         /dev/sda1: using cached size 1024000 sectors
#cache/lvmetad.c:2247          Scan metadata from dev /dev/sda1
#cache/lvmetad.c:2260      No PV info found on /dev/sda1 for PVID .
#cache/lvmetad.c:1746          Telling lvmetad to forget any PV on /dev/sda1
#cache/lvmetad.c:784         Request to drop PV /dev/sda1 in lvmetad did not find any matching object.
#device/dev-io.c:336         /dev/sda2: using cached size 1047549952 sectors
#filters/filter-mpath.c:196           /dev/sda2: Device is a partition, using primary device sda for mpath component detection
#device/dev-io.c:336         /dev/sda2: using cached size 1047549952 sectors
#cache/lvmetad.c:2247          Scan metadata from dev /dev/sda2
#cache/lvmetad.c:2260      No PV info found on /dev/sda2 for PVID .
#cache/lvmetad.c:1746          Telling lvmetad to forget any PV on /dev/sda2
#cache/lvmetad.c:784         Request to drop PV /dev/sda2 in lvmetad did not find any matching object.
#device/dev-io.c:336         /dev/sdb: using cached size 4294967296 sectors
#filters/filter-partitioned.c:37            /dev/sdb: Skipping: Partition table signature found
#device/dev-io.c:336         /dev/sdb1: using cached size 4294965247 sectors
#filters/filter-mpath.c:196           /dev/sdb1: Device is a partition, using primary device sdb for mpath component detection
#device/dev-io.c:336         /dev/sdb1: using cached size 4294965247 sectors
#cache/lvmetad.c:2247          Scan metadata from dev /dev/sdb1
#cache/lvmetad.c:2260      No PV info found on /dev/sdb1 for PVID .
#cache/lvmetad.c:1746          Telling lvmetad to forget any PV on /dev/sdb1
#cache/lvmetad.c:784         Request to drop PV /dev/sdb1 in lvmetad did not find any matching object.
#device/dev-io.c:336         /dev/sdc: using cached size 20971520 sectors
#filters/filter-partitioned.c:37            /dev/sdc: Skipping: Partition table signature found
#device/dev-io.c:336         /dev/sdc1: using cached size 2097152 sectors
#filters/filter-mpath.c:196           /dev/sdc1: Device is a partition, using primary device sdc for mpath component detection
#device/dev-io.c:336         /dev/sdc1: using cached size 2097152 sectors
#cache/lvmetad.c:2247          Scan metadata from dev /dev/sdc1
#cache/lvmetad.c:2260      No PV info found on /dev/sdc1 for PVID .
#cache/lvmetad.c:1746          Telling lvmetad to forget any PV on /dev/sdc1
#cache/lvmetad.c:784         Request to drop PV /dev/sdc1 in lvmetad did not find any matching object.
#device/dev-io.c:336         /dev/sdc2: using cached size 2097152 sectors
#filters/filter-mpath.c:196           /dev/sdc2: Device is a partition, using primary device sdc for mpath component detection
#device/dev-io.c:336         /dev/sdc2: using cached size 2097152 sectors
#cache/lvmetad.c:2247          Scan metadata from dev /dev/sdc2
#cache/lvmetad.c:2260      No PV info found on /dev/sdc2 for PVID .
#cache/lvmetad.c:1746          Telling lvmetad to forget any PV on /dev/sdc2
#cache/lvmetad.c:784         Request to drop PV /dev/sdc2 in lvmetad did not find any matching object.
#device/dev-io.c:336         /dev/sdc3: using cached size 16775168 sectors
#filters/filter-mpath.c:196           /dev/sdc3: Device is a partition, using primary device sdc for mpath component detection
#device/dev-io.c:336         /dev/sdc3: using cached size 16775168 sectors
#cache/lvmetad.c:2247          Scan metadata from dev /dev/sdc3
#cache/lvmetad.c:2260      No PV info found on /dev/sdc3 for PVID .
#cache/lvmetad.c:1746          Telling lvmetad to forget any PV on /dev/sdc3
#cache/lvmetad.c:784         Request to drop PV /dev/sdc3 in lvmetad did not find any matching object.
#cache/lvmetad.c:558           Sending lvmetad token_update filter:3239235440
#vgscan.c:118     Reading volume groups from cache.
#toollib.c:2246          Processing each VG
#toollib.c:2294        Obtaining the complete list of VGs to process
#cache/lvmetad.c:1447          Asking lvmetad for complete list of known VG ids/names
#toollib.c:2314      No volume groups found.
#misc/lvm-flock.c:70          Unlocking /run/lock/lvm/P_global
#misc/lvm-flock.c:47            _undo_flock /run/lock/lvm/P_global
#cache/lvmcache.c:2539          Dropping VG info
#cache/lvmcache.c:755           lvmcache has no info for vgname "#orphans_lvm1" with VGID #orphans_lvm1.
#cache/lvmcache.c:755           lvmcache has no info for vgname "#orphans_lvm1".
#cache/lvmcache.c:2086          lvmcache: Initialised VG #orphans_lvm1.
#cache/lvmcache.c:755           lvmcache has no info for vgname "#orphans_pool" with VGID #orphans_pool.
#cache/lvmcache.c:755           lvmcache has no info for vgname "#orphans_pool".
#cache/lvmcache.c:2086          lvmcache: Initialised VG #orphans_pool.
#cache/lvmcache.c:755           lvmcache has no info for vgname "#orphans_lvm2" with VGID #orphans_lvm2.
#cache/lvmcache.c:755           lvmcache has no info for vgname "#orphans_lvm2".
#cache/lvmcache.c:2086          lvmcache: Initialised VG #orphans_lvm2.
#lvmcmdline.c:3050          Completed: vgscan -vvvv

closed as off-topic by Christopher, Mr Shunz, jimmij, Archemar, Thomas Feb 9 at 9:30

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Questions describing a problem that can't be reproduced and seemingly went away on its own (or went away when a typo was fixed) are off-topic as they are unlikely to help future readers." – Christopher, Mr Shunz, jimmij, Archemar, Thomas
If this question can be reworded to fit the rules in the help center, please edit the question.

  • The question says, "The O/S on the broken drives is RHEL 7." – G-Man Feb 2 at 1:03
  • Try hexdump -C | less instead of dd. Your vgscan seems to be only looking at cache, try without lvmetad (kill it if necessary). LVM might also have backups of metadata in /etc/lvm/.... – frostschutz Feb 2 at 1:15
  • Thanks @frostschutz , post updated withoutusing cache - least there is info about the drive now! I was not successful with the hexdump, it complained that the line is too long. – Werner Feb 2 at 4:09
  • @G-Man, apologies if it is not clear, the server is running RHEL 7, I added the 10G drive and installed CentOS off that to have a working O/S to troubleshoot from. – Werner Feb 2 at 4:28
  • (1) Now I’m confused.  Do you mean “the server was running RHEL 7 before it failed”?  (2) If the question is ambiguous or unclear, please edit the question to fix it.  People who are confused by the main body of the question won’t bother reading all the comments in a search for clarifying clues. (3) I wasn’t confused or asking a question.  Somebody posted a comment asking what OS the machine was running.  I posted my comment quoting the question, and they deleted their comment. – G-Man Feb 2 at 5:47
1

At this stage we suspect it was a side effect of a process ran by the storage teams. They ran a process that would ‘zero unlinked blocks‘. So our guess is that process caused some destruction.

Not the answer you're looking for? Browse other questions tagged or ask your own question.