Attempted to replicate partition table to replacement RAID1 disk, but source partition table now empty

Question

I had a disk (/dev/sda) show signs of eventual failure in a RAID1 array, so I failed and then removed it from the array.

I then replaced the disk, booted back up, and began the process of replicating the partition in order to add the disk to the array, however something went wrong.

The final command I used was:

sgdisk -R /dev/sdb /dev/sda

Now lsblk shows the correct partitioning for /dev/sdb:

[root@server /]# lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
nvme0n1     259:1    0   477G  0 disk  
├─nvme0n1p3 259:4    0   7.8G  0 part  
│ └─md3       9:3    0   7.8G  0 raid1 /tmp
├─nvme0n1p1 259:2    0   511M  0 part  /boot/efi
├─nvme0n1p4 259:5    0   7.8G  0 part  [SWAP]
└─nvme0n1p2 259:3    0 460.8G  0 part  
  └─md2       9:2    0 460.8G  0 raid1 /
sdb           8:16   0   3.7T  0 disk  
└─sdb1        8:17   0   3.7T  0 part  
  └─md4       9:4    0   3.7T  0 raid1 /var
nvme1n1     259:0    0   477G  0 disk  
├─nvme1n1p4 259:9    0   7.8G  0 part  [SWAP]
├─nvme1n1p2 259:7    0 460.8G  0 part  
│ └─md2       9:2    0 460.8G  0 raid1 /
├─nvme1n1p3 259:8    0   7.8G  0 part  
│ └─md3       9:3    0   7.8G  0 raid1 /tmp
└─nvme1n1p1 259:6    0   511M  0 part  
sda           8:0    0   3.7T  0 disk  

However, sda does not show the same, but worse, when I run:

sgdisk -p /dev/sdb

It does not show me a partition table, same for /dev/sda:

[root@server dev]# sgdisk -p /dev/sdb
Disk /dev/sdb: 7814037168 sectors, 3.6 TiB
Logical sector size: 512 bytes
Disk identifier (GUID): 34DA93D9-0A46-433D-BDE3-6AF2566E2183
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 7814037134
Partitions will be aligned on 2048-sector boundaries
Total free space is 7814037101 sectors (3.6 TiB)

Number  Start (sector)    End (sector)  Size       Code  Name
[root@server dev]# sgdisk -p /dev/sda
Disk /dev/sda: 7814037168 sectors, 3.6 TiB
Logical sector size: 512 bytes
Disk identifier (GUID): EBADBC60-3D20-48F7-880B-5CCF1B645A44
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 7814037134
Partitions will be aligned on 2048-sector boundaries
Total free space is 7814037101 sectors (3.6 TiB)

Number  Start (sector)    End (sector)  Size       Code  Name
[root@server dev]# 

When I ran partprobe, it gave me the following error:

[root@server dev]# partprobe
Error: Partition(s) 1 on /dev/sdb have been written, but we have been unable to inform the kernel of the change, probably because it/they are in use.  As a result, the old partition(s) will remain in use.  You should reboot now before making further changes.

Now I am not the most experienced linux administrator (obviously), but I am guessing that somehow instead of duplicating /dev/sdb to /dev/sda I actually did the reverse and cleared the partition table for /dev/sdb.

Thankfully I have not rebooted the machine, so the system is live and functioning and I would expect that there would be some way to recover the working partition table?

Now the big unfortunate is that this is a production server and it going down/offline for an extended period of time would be pretty devastating. So I'm hoping someone could guide me through getting this back to normal.

I'm not sure what else to share here to get help, so feel free to ask me to post results of anything.

Thanks in advance.

Answer 1

If the kernel still knows the correct partition table, you can query partition start offsets and sizes like this:

# partition start offsets
head /sys/block/sdf/sdf*/start

# partition sizes
head /sys/block/sdf/sdf*/size

Sample output:

$ head /sys/block/sdf/sdf*/start
==> /sys/block/sdf/sdf1/start <==
2048

==> /sys/block/sdf/sdf2/start <==
4198400

==> /sys/block/sdf/sdf3/start <==
8394752

==> /sys/block/sdf/sdf4/start <==
64

$ head /sys/block/sdf/sdf*/size
==> /sys/block/sdf/sdf1/size <==
4194304

==> /sys/block/sdf/sdf2/size <==
4194304

==> /sys/block/sdf/sdf3/size <==
52166656

==> /sys/block/sdf/sdf4/size <==
1984

What it looks like in parted:

# parted /dev/sdf unit s print
Model:  Patriot Memory (scsi)
Disk /dev/sdf: 60566016s
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: pmbr_boot

Number  Start     End        Size       File system  Name     Flags
 4      64s       2047s      1984s                   grub     bios_grub
 1      2048s     4196351s   4194304s   fat32        freedos  msftdata
 2      4198400s  8392703s   4194304s   ext2         boot     lvm
 3      8394752s  60561407s  52166656s  ext2         iso      lvm

That way, you can easily re-create partitions at the correct offsets.

If there are any special partition flags (bios_grub, boot, esp, ...) they have to be provided manually, but in your case that seems to be on SSD and the HDD just has a simple data partition, so one less thing to worry about.

Since your /dev/sdb only has a single partition /dev/sdb1, chances are it will start at 1 MiB and extend to the full size of the disk. So re-creating that one partition couldn't be any simpler. Still, it's still good to double check.

Alternatively, you could use testdisk to deduce the partition table from raw data.