Access ext4 disk that lost superblock and its backups

Question

I found myself in a weird situation with my system's disk. It's Linux system with 3 partitions, boot, swap and filesystem (ext4). A few days ago i used knoppix from a pendrive (I made the bootable usb drive from a knoppix image myself) because i wanted to check an unreadable disk...and i found myself with 2 unreadable disks!

When i rebooted the system i started getting on my screen errors like this:

error: failure reading sector ... from 'hd0'.

and then i got the grub shell after a few attempts.

I created another bootable stick with Linux Mint and i tried to see what went wrong.
When checking the disk using the "disks" tool it said the disk was ok but has 8 bad sectors (i'm not sure they were there before), the filesystem partition was there but the type for the data partition was unkown.
I tried running testdisk. It found the partitions, and once i checked the superblocks and set the Filesystem type (ext4) i could see the data from testdisk itself but still could not mount the partition.
Testdisk suggested to run e2fsck -p -b and i tried with all the superblocks testdisk gave me but every time fsck tried to fix something it ended with a disk write error.
At this point it was late, so i gave up for the night, but kinda reassured that the data was still readable
The day after with my suprise the data was not there anymore, probably some run of fsck made things worse because testdisk would not give me any superblock number anymore (and i couldn't see the files from testdisk anymore no matter what i tried), and if i tried running fsck with the numbers i got previously (i had saved them) i got Bad magic number in super-block while trying to open /dev/sda3 no matter the number
At this point i made an image with safecopy (which i admit, i shouldve done as first thing but I was a bit panicked so i didn't think about that) and duplicated it so that i could test different solutions, making a new copy from the starting image every time i reached a point i didn't like
All the things i tried did not help tho. I found a suggestion to use mkfs with the -S option so that it would only rebuild the superblocks, but when i tried that and ran fsck on it afterwards (as suggested by the tool) it would give me access to the partition again, but the partition would be completely empty. So empty that not even photorec could find data anymore (it would find data on the image before running mkfs + fsck).

I ran R-Linux on one of the images but it couldn't recover anything, all i had are "files" named $Inode..., but no discernable file or directory structure.

This is the result of smartctl -a /dev/sda

smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.4.0-58-generic] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Toshiba 2.5" HDD MQ01ABD...
Device Model:     TOSHIBA MQ01ABD100
Serial Number:    95CEC91QT
LU WWN Device Id: 5 000039 683983c2b
Firmware Version: AX0R2J
User Capacity:    1,000,204,886,016 bytes [1.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Form Factor:      2.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Tue Jun  1 22:30:50 2021 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                    was never started.
                    Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                    without error or no self-test has ever 
                    been run.
Total time to complete Offline 
data collection:        (  120) seconds.
Offline data collection
capabilities:            (0x5b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    No Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                    General Purpose Logging supported.
Short self-test routine 
recommended polling time:    (   2) minutes.
Extended self-test routine
recommended polling time:    ( 246) minutes.
SCT capabilities:          (0x003d) SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   050    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0005   100   100   050    Pre-fail  Offline      -       0
  3 Spin_Up_Time            0x0027   100   100   001    Pre-fail  Always       -       1815
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       1377
  5 Reallocated_Sector_Ct   0x0033   100   100   050    Pre-fail  Always       -       8
  7 Seek_Error_Rate         0x000b   100   100   050    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   100   100   050    Pre-fail  Offline      -       0
  9 Power_On_Hours          0x0032   053   053   000    Old_age   Always       -       19115
 10 Spin_Retry_Count        0x0033   127   100   030    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       1368
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       275
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       50
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       4455
194 Temperature_Celsius     0x0022   100   100   000    Old_age   Always       -       30 (Min/Max 13/49)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       1
197 Current_Pending_Sector  0x0032   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       1
220 Disk_Shift              0x0002   100   100   000    Old_age   Always       -       0
222 Loaded_Hours            0x0032   053   053   000    Old_age   Always       -       18889
223 Load_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
224 Load_Friction           0x0022   100   100   000    Old_age   Always       -       0
226 Load-in_Time            0x0026   100   100   000    Old_age   Always       -       265
240 Head_Flying_Hours       0x0001   100   100   001    Pre-fail  Offline      -       0

SMART Error Log Version: 1
ATA Error Count: 2032 (device log contains only the most recent five errors)
    CR = Command Register [HEX]
    FR = Features Register [HEX]
    SC = Sector Count Register [HEX]
    SN = Sector Number Register [HEX]
    CL = Cylinder Low Register [HEX]
    CH = Cylinder High Register [HEX]
    DH = Device/Head Register [HEX]
    DC = Device Command Register [HEX]
    ER = Error register [HEX]
    ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 2032 occurred at disk power-on lifetime: 19112 hours (796 days + 8 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 41 38 00 d8 16 40  Error: UNC at LBA = 0x0016d800 = 1497088

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 08 38 00 d8 16 40 00   1d+05:06:46.679  READ FPDMA QUEUED
  ef 10 03 00 00 00 a0 00   1d+05:06:46.678  SET FEATURES [Enable SATA feature]
  ef 10 02 00 00 00 a0 00   1d+05:06:46.678  SET FEATURES [Enable SATA feature]
  27 00 00 00 00 00 e0 00   1d+05:06:46.678  READ NATIVE MAX ADDRESS EXT [OBS-ACS-3]
  ec 00 00 00 00 00 a0 00   1d+05:06:46.677  IDENTIFY DEVICE

Error 2031 occurred at disk power-on lifetime: 19112 hours (796 days + 8 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 41 68 00 d8 16 40  Error: UNC at LBA = 0x0016d800 = 1497088

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 08 68 00 d8 16 40 00   1d+05:06:45.909  READ FPDMA QUEUED
  e5 00 00 00 00 00 00 00   1d+05:06:45.669  CHECK POWER MODE
  e5 00 00 00 00 00 00 00   1d+05:06:40.669  CHECK POWER MODE
  e5 00 00 00 00 00 00 00   1d+05:06:35.670  CHECK POWER MODE
  e5 00 00 00 00 00 00 00   1d+05:06:30.669  CHECK POWER MODE

Error 2030 occurred at disk power-on lifetime: 19110 hours (796 days + 6 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 41 30 00 d8 16 40  Error: UNC at LBA = 0x0016d800 = 1497088

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 00 e8 fd de 40 00   1d+02:46:45.118  READ FPDMA QUEUED
  60 00 f8 e8 fc de 40 00   1d+02:46:45.117  READ FPDMA QUEUED
  60 00 b8 e8 fb de 40 00   1d+02:46:45.116  READ FPDMA QUEUED
  60 00 b0 e8 fa de 40 00   1d+02:46:45.116  READ FPDMA QUEUED
  60 00 a8 e8 f9 de 40 00   1d+02:46:45.115  READ FPDMA QUEUED

Error 2029 occurred at disk power-on lifetime: 19110 hours (796 days + 6 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 41 a0 00 d8 16 40  Error: UNC at LBA = 0x0016d800 = 1497088

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 08 e8 f8 de 40 00   1d+02:46:44.968  READ FPDMA QUEUED
  60 00 00 e8 f7 de 40 00   1d+02:46:44.968  READ FPDMA QUEUED
  60 00 f0 e8 f6 de 40 00   1d+02:46:44.966  READ FPDMA QUEUED
  60 00 e8 e8 f5 de 40 00   1d+02:46:44.965  READ FPDMA QUEUED
  60 00 e0 e8 f4 de 40 00   1d+02:46:44.964  READ FPDMA QUEUED

Error 2028 occurred at disk power-on lifetime: 19110 hours (796 days + 6 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 41 88 00 d8 16 40  Error: UNC at LBA = 0x0016d800 = 1497088

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 a0 e8 e8 de 40 00   1d+02:46:44.792  READ FPDMA QUEUED
  60 00 98 e8 e7 de 40 00   1d+02:46:44.791  READ FPDMA QUEUED
  60 00 90 e8 e6 de 40 00   1d+02:46:44.788  READ FPDMA QUEUED
  60 00 78 e8 e5 de 40 00   1d+02:46:44.787  READ FPDMA QUEUED
  60 00 70 e8 e4 de 40 00   1d+02:46:44.786  READ FPDMA QUEUED

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

What does smartctl -a /dev/sda return, given /dev/sda is the faulty disk — Chris Davies, Jun 1, 2021 at 20:29
At a guess I'd say the disk was around 3½ years old. It's had 8 failed sectors that were reallocated on a fresh write of newer data (such errors can only get resolved when you try to write new data to the sector). You might have lost the beginning of the partition where the primary superblock lives but you should still be able to recover it from a backup — Chris Davies, Jun 1, 2021 at 21:09
Have you actually run all these things you've listed? Item 6 may have either worked or corrupted the backup superblocks, depending on too many factors to list here. Item 5 might give you a chance at recovery. Maybe. Depending on how many errors it found. Did you safecopy the partition (sda3) or the disk (sda)? — Chris Davies, Jun 1, 2021 at 21:13
yes i did run all these things. i used safecopy on the whole disk but i ran mkfs on the partition (mounted with losetup). The first time i ran mkfs on the disk (mkfs -S /dev/sda) and it removed all the partitions. I am confident i used -S — valepu, Jun 1, 2021 at 21:15

valepu · Accepted Answer · 2021-07-12 19:41:28Z

In the end i realized that Knoppix had messed up with my partition table when i used the live version from a USB stick, the disk was actually ok (except for some bad sectors), I tried many ways to recover the proper partitions but to no avail.
As i said in my question and in several comments, using mkfs allowed me to mount the partition back but with no data and apparently this seems to be the only way to rebuild superblocks, but i couldn't find a proper way to do it.

After 2 weeks of attempts I decided to format everything and start from scratch since all the important data I needed is stored inside a repository anyways.

Chris Davies · Accepted Answer · 2021-06-01 21:40:41Z

-1

ALWAYS but ALWAYS read the documentation before just doing something blindly. man mkfs.ext4 and search for the -S flag, where it will tell you that you must run e2fsck afterwards.

I suspect if you've written to your saved copy of the disk then this is now also dead. Rule 1 of disk recovery is to take a copy as soon as possible. Rule 2 is NEVER to write to your saved copy.

One thing I have noticed is that you're confusing the disk (/dev/sda) with the partition (/dev/sda3). Where you are sure you ran mkfs -S /dev/sda you've tried to recreate a filesystem on the disk rather than the partition. If you can restore the partition table you might be able to run mkfs -S /dev/sda3 successfully. Or equivalents via the loop device.

NOTE: Please do not try writing to both your dying disk and your only saved copy anymore. Take a fresh copy of the copy and work on that.

answered Jun 1, 2021 at 21:40

Chris Davies

111k14 gold badges150 silver badges275 bronze badges

I think you are misunderstanding me: 1. I am aware that i must run e2fsck afterwards. I have done so everytime i ran mkfs (the first time i was told to do it by the disks tool from Linux mint) but when i run e2fsck i get the partition back but it's empty! 2. as i said. I ran mkfs -S /dev/sda the first time (on the image) but it wiped the disk, so i took another copy of the image and ran mkfs -S /dev/sda3 afterwards (on a fresh copy of the image).
– valepu
Jun 1, 2021 at 21:45
3. As I said in my question, i have 2 copies of the image. I run stuff on one image, then when it doesn't work i delete the image and make a new copy from the first image. After i created the image i haven't touched the disk anymore
– valepu
Jun 1, 2021 at 21:48
That's good to hear. I've just re-read your question and I still can't see anywhere that says you work on copies of your original copy. It might be useful to review your comments and incorporate the appropriate ones into your question - you can then delete the comments. If no-one else's comment is then relevant anymore you can flag the first with the moderators for deletion, ask them to delete them all because you've addressed them in your question. This way everything's in a single place and easy to read - and hopefully easy for someone (else) to answer.
– Chris Davies
Jun 1, 2021 at 21:55
I say it on point 5. At this point i made an image with safecopy (which i admit, i shouldve done as first thing but I was a bit panicked so i didn't think about that) and duplicated it so that i could test different solutions. It's not explicit but when i say "duplicated" i was implying i made a copy for every test
– valepu
Jun 1, 2021 at 21:59
3

@roaima According to the history, the first version does make this explicit. Notwithstanding sub-five-minute edits, I don't think its fair to claim that this was only mentioned at the time your answer was written.
– Philip Couling
Jun 3, 2021 at 13:42

| Show 2 more comments

Stack Exchange Network

Access ext4 disk that lost superblock and its backups

2 Answers 2

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged
hard-disk
fsck
superblock
.

Hot Network Questions

Access ext4 disk that lost superblock and its backups

2 Answers 2

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged hard-diskfscksuperblock.

Related

Hot Network Questions

Not the answer you're looking for? Browse other questions tagged
hard-disk
fsck
superblock
.