Adding badblocks on mounted partition

Question

I tried to move some files to my NAS (ShareCenter DNS-320), but something shows up, when using file managers:

Input/Output error

or when using rsync on mounted cifs/smb share

rsync: close failed on "/mnt/nas1/_am-unordered/.long-file-name.mkv.PI2rPM": Input/output error (5)
rsync error: error in file IO (code 11) at receiver.c(856) [receiver=3.1.0]

# mount | grep mnt/nas1

//192.168.x.y/backup on /mnt/nas1 type cifs (rw,relatime,vers=1.0,cache=strict,username=admin,domain=BACKUP,uid=1000,forceuid,gid=0,noforcegid,addr=192.168.x.y,file_mode=0755,dir_mode=0755,nounix,serverino,rsize=61440,wsize=65536,actimeo=1)

I assume that there are bad sectors inside the NAS, I need to run fsck to check if there are broken disk inside my RAID-0 NAS.

I have installed fun_plug using this tutorial, now I could ssh into the NAS successfully. Normally I would use fsck -yvckfC -E fragcheck /dev/sdX to check bad sectors on single unmounted disk.

The question is, how to run badblocks and insert it to the bad block list on mounted RAID0 partition? since the ssh service are running on mounted partition on the NAS:

# umount /mnt/HD/HD_a2/
  umount: /mnt/HD/HD_a2: device is busy.
    (In some cases useful info about processes that use
     the device is found by lsof(8) or fuser(1))

# lsof /mnt/HD/HD_a2/
COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF      NODE NAME
sh        1963 root    1w   REG    9,1     5191        12 /mnt/HD/HD_a2/ffp.log
sh        1963 root    2w   REG    9,1     5191        12 /mnt/HD/HD_a2/ffp.log
sh        1963 root   10r   REG    9,1     1942 246939802 /mnt/HD/HD_a2/fun_plug
rc        1986 root  txt    REG    9,1   587316 141426950 /mnt/HD/HD_a2/ffp/bin/bash
rc        1986 root  mem    REG    9,1    28892 139854377 /mnt/HD/HD_a2/ffp/lib/ld-uClibc-0.9.33-git.so
rc        1986 root  mem    REG    9,1   260898 139853932 /mnt/HD/HD_a2/ffp/lib/libreadline.so.6.2
 **snip**
sshd      5519 root  mem    REG    9,1    60546 139854375 /mnt/HD/HD_a2/ffp/lib/libgcc_s.so.1
sshd      5519 root  mem    REG    9,1   359940 139854378 /mnt/HD/HD_a2/ffp/lib/libuClibc-0.9.33-git.so

Current NAS' RAID configuration are:

# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] 
md1 : active raid0 sda2[0] sdb2[1]
      7808789888 blocks 64k chunks

md0 : active raid1 sdb1[1] sda1[0]
      524224 blocks [2/2] [UU]

unused devices: <none>

Answer 1

You're working from a shaky premise, being that badblocks can solve your problem in the first place.

Why badblocks Is an Untrustworthy Repair Method

As you use a hard drive, it continually does its best to hide problems from you by swapping fresh sectors in for dodgy ones. The hard disk ships from the factory with a pool of spare sectors for this very purpose. As long as the number of new bad sectors grows slowly, the spare sector pool shrinks slowly enough that the hard drive appears to run flawlessly.

The only way badblocks can detect a bad sector is when the spare sector pool has run dry, which means that it has been degrading for some time. Put another way, visible bad sectors mean the hard disk has swept so many problems under the rug that the rug is starting to look lumpy.

As far as I'm aware, hard drives have done this sort of silent fix for decades now, probably from the early days of IDE. The last systems I used that exposed their initial set of bad sectors from the start used ESDI and MFM hard disks, dating from the late 1980s.

This is not to say that modern hard drives no longer ship with an initial set of bad sectors. They do. Bad sectors are mapped out at the factory so that a badblocks test on a new hard disk will turn up zero bad sectors. (Sectors from the spare sector pool are mapped in to take the bad sectors' place.)

If a badblocks scan turns up bad sectors on a new drive or one still within its warranty period, that's sufficient reason to have it replaced immediately.

It is possible for badblocks to return a consistent result over a long enough period of time for the filesystem's bad sector list to be useful. This can indeed allow you to continue using an out-of-warranty or otherwise irreplaceable hard drive past the point where the drive's own self-repair features have stopped working.

However, it is also possible for badblocks to return different results between closely spaced tests. (Say, two tests done a day or a week apart.) When the hard disk gets into such a bad state, the filesystem's bad sector list becomes pointless; the hard disk is dying. The filesystem's bad sector list only provides a benefit when the list remains stable over long periods of time.

Bottom line: Replace the hard disk while it is still readable. Yes, I realize this probably means rebuilding the entire NAS, but that's the cost of RAID-0, a.k.a. "scary RAID."

A Better Solution: Monitoring

You cannot tell that a sector swap has taken place short of tracking the size of the spare sector pool over time via SMART. Some hard drives won't report this, even if you did want to track it, and those that do provide it may report only a modified version of the truth rather than the literal truth.

That said, this command may tell you what you need to know:

# smartctl -x /dev/sda | grep Realloc
  5 Reallocated_Sector_Ct   PO--CK   200   200   140    -    0
196 Reallocated_Event_Count -O--CK   200   200   000    -    0

While the raw and normalized values that smartctl reports may not be precisely correct, an increasing number here — especially a large increase over a brief period — is always bad.

Notice that the last column is zero on the machine I ran that command on. This is what I mean when I say the report may not be entirely trustworthy. That's the "raw" value, while the "200" columns are the "normalized" value. This drive is claiming there have been no reallocations, ever, which is almost certainly not true. As for "200", that is a value the hard drive manufacturer came up with on their own, with their own meaning. You can't compare it between hard drive brands, and may not even be able to compare it to other hard drives from the same manufacturer.

But once again: if you monitor these values and they suddenly start increasing, that is a bad sign, even though it doesn't actually tell you what's going on at the oxide level.

smartctl reports information on individual hard drives, not RAID devices. It does know how to talk to several types of hardware RAID controllers to extract per-drive information, but there's no need to have specific support for software RAID, since the underlying devices are directly available. Thus, you'd need to monitor both /dev/sda and /dev/sdb separately in your case, not /dev/md1.

smartd — a companion tool to smartctl — does this sort of background continuous monitoring.