Avoid damaged block in ext4

Question

I am running

$ uname -a
Linux myhostname 4.14.15-041415-generic #201801231530 SMP Tue Jan 23 20:33:21 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
$ lsb_release -a
No LSB modules are available.
Distributor ID: Nitrux
Description:    Nitrux 1.1.4
Release:        1.1.4
Codename:       nxos

It has a single hard disk with a system ext4 partition and a swap partition. The hard disk can't complete neither the Smart short test, nor the long one.

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed: read failure       90%     32232         11202419
# 2  Extended offline    Completed: read failure       90%     32229         11202419

Maybe the disk should be replaced.

In the meanwhile, is it possible to simply instruct the filesystem to avoid the block corresponding to that LBA? So that no further read/write errors are generated from there. In fact, it seems to be an isolated error and the hard disk (except, of course, for that area) is still able to work.

---

The SMART parameters are weird, because there are pending sectors to be re-allocated, but there are also 0 reallocated sectors. Note that this hard disk is about 10 years old.

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       19
  3 Spin_Up_Time            0x0027   140   139   021    Pre-fail  Always       -       3966
  4 Start_Stop_Count        0x0032   098   098   000    Old_age   Always       -       2058
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   056   056   000    Old_age   Always       -       32232
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   098   098   000    Old_age   Always       -       2001
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       206
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       1851
194 Temperature_Celsius     0x0022   103   086   000    Old_age   Always       -       40
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       78
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       70
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       89

---

In the linked page there is no chosen answer. I must keep the system up and I would like to avoid dd (and there is no clear example about how to use it in this case). Can I run fsck.ext2 -c on a mounted filesystem?

Answer 1

From the e2fsck man page (e2fsck is also linked to names fsck.ext2, fsck.ext3 and fsck.ext4):

Note that in general it is not safe to run e2fsck on mounted filesystems. The only exception is if the -n option is specified, and -c, -l, or -L options are not specified. However, even if it is safe to do so, the results printed by e2fsck are not valid if the filesystem is mounted. If e2fsck asks whether or not you should check a filesystem which is mounted, the only correct answer is "no". Only experts who really know what they are doing should consider answering this question in any other way.

So the answer is "no, you cannot run fsck on a mounted ext2/3/4 filesystem in any mode that would make any changes to the filesystem at all".

At boot time, the root filesystem may be checked while it's mounted in read-only mode or the system is still running on initramfs. But in this situation, the system should be rebooted immediately afterwards if the fsck indicates it had to make any changes.

---

If a disk block has totally failed so that even repeated retries won't result in the disk being confident that the data has been read correctly, the disk cannot automatically reallocate that block until its contents are overwritten by the OS - because doing the reallocation without having the correct data is equivalent to silently corrupting the data (by replacing a block of data with zeroes). That is worse than a file that simply produces a read error, because the corrupt data may be used in further processing and silently cause other results to be corrupted until it is finally noticed.

A file that produces read errors is usually pretty straightforward to restore from backups, unless it is a critical system file and the system crashes or is unable to run the restore tool if that file is missing.

The fact that SMART indicates there are sectors pending to be re-allocated but no actual re-allocations might mean just that the failed sectors are occupied by system files that are normally only ever read and practically never written. If you can figure out which package those files belong to, you can instruct the package management system to reinstall that package; Nitrux seems to use .deb packages, so apt-get reinstall <package name> would be the command to run. This would cause the file to be rewritten, allowing the disk to complete the re-allocation.

Unfortunately, some disk manufacturers have created disks with incomplete SMART implementations, so you can only really trust SMART if it's telling bad news; if it says things are OK but the operating system is reporting read/write errors, then something is bad regardless of what SMART says - and since HDDs are a wear item, in most cases it's the disk that is faulty.

---

I've worked in various roles in server administration for a living for more than 20 years now. Through all that time, our team's reaction on seeing a 10+ years old disk still in use would have been - and still is:

"Holy ****! If that disk spins down for any reason, there is practically no guarantee at all that it will ever restart again. Can we even get spare parts for old hardware like that with any reasonable price and response time? At the very least, we need to make a very realistic plan on what to do when (not if) that thing fails, preferably get the ball rolling right now on either replacement or virtualization of that old thing ASAP."

Granted, we deal with servers that are almost always running 24 hours a day, every day of every year, through their whole lifetime - and that might not be the case with your system.

But a 10 year old disk, if used anywhere near the "typical" way for the market segment it's designed for, is definitely well into the rising edge of the bathtub curve: its design lifetime has been exceeded and it's wearing out.