2

I have an application that write files into a specific directory at the FS. This system, under no circumstances, can lose a single written file.

When I have some problem at the HD, like badblocks or blocks bitmap differences, new and even existing files might get corrupted, but the rsync and the .tar backup happen normally and I'll know that the file has a problem only when I try to open it.

I need a way to know that a file get corrupted as soon as it happens or, better yet, know the disk has a problem before it brakes my files.

I was thinking to keep a file with the hash of each file and check it every day, but it will spend more time than I have. Another idea would be put those files into the SGDB, but I wonder if there is a better way to solve this.

I'm pretty sure that I'm missing something, but I'm blind.

System: Debian 6 and Debian 7 32bit and 64bit (the application is installed in many places). All systems are ext4 (is there any other more trustful?)

Improve this question
6
  • 2
    It looks like ZFS might be something for you. It does automatic checksumming and can correct errors (if you have redunadancy, which you totally should). Furthermore, the snapshots help a lot in setting up a backup system including history.
    – Marco
    Commented Apr 28, 2015 at 11:50
  • What is "SGDB"?
    – Bratchley
    Commented Apr 28, 2015 at 13:17
  • 1
    @Bratchley I'm guessing it's SGBD, which means DBMS in French (système de gestion de bases de données).
    – Stephen Kitt
    Commented Apr 28, 2015 at 13:23
  • 1
    Since, as you seem to say, have already severe (unrecoverable?) consistency problems with your current file system it's high time to fix that; replace your disks and best install a new file system that supports all that safety. (This is what I did after I had such issues with ext4 and an unreliable hard disk with unrecoverable errors; I installed ZFS in "raidz" mode. After a week the unreliable harddisk gave up, and I could still continue to work (in "degraded" state), and it was easy to replace the defect disk and ZFS incorporated the new disk without problem.)
    – Janis
    Commented Apr 28, 2015 at 13:25
  • Sorry by de SDGB, its DBMS in english
    – slacker
    Commented Apr 28, 2015 at 13:48

4 Answers 4

Reset to default
4

If one could that easily detect when sectors are about to go bad or do go bad, it would likely have been worked into the filesystem by now. Due to the nature of the error, it will often be silent. You need a filesystem that does checksumming. On GNU/Linux BTRFS may be a good bet since I looked online and apparently support was introduced in Debian 6.

Basically you need checksumming+raid (of some kind). A filesystem can only autocorrect if it has at least two legs in a RAID setup. Without a second leg to go to it has no where to go in order to find a verifiably good copy of the file. Luckily you should be able to create a RAID1 array using two different partitions on the same disk (or logical volumes, whatever you happen to have available):

enter image description here

Obviously, putting them on the same disk doesn't protect against a completely failed disk but it protects against failed sectors. Simulating a failed sector is probably more work than I should put into an SE answer but this guy (test starts at the 24:30 mark) runs through a demonstration for you.

Basically, BTRFS will opaquely recover the file and the user space will not be aware anything has happened. You can use the btrfs scrub to detect errors after the fact. You can run that in a cronjob and have it email one of you local accounts. After that you can set up /etc/aliases to forward the command output to your real email account.

Improve this answer
3

You can install and configure the SMART monitoring tools. On Debian the package is called smartmontools. These won't prevent disk failure but they will help identify precursors to possible disk failure.

There is no configuration in the package installation, so you need first to enable SMART monitoring in the file /etc/default/smartmontools:

# uncomment to start smartd on system startup
start_smartd=yes

and then edit the configuration file /etc/smartd.conf:

# The word DEVICESCAN will cause any remaining lines in this
# configuration file to be ignored [...]
# [...] Most users should comment out DEVICESCAN and explicitly
# list the devices that they wish to monitor.
#DEVICESCAN -d removable -n standby -m root -M exec /usr/share/smartmontools/smartd-runner


# Short test nightly, Long test on Sunday mornings; append "-m [email protected]" to email errors
/dev/sda -a -s (S/../.././02|L/../../6/03)
/dev/sdb -a -s (S/../.././04|L/../../6/05)
/dev/sdc -a -s (S/../.././06|L/../../6/07)
# /dev/sdd -a -s (S/../.././06|L/../../6/07) -m [email protected]

Finally start the monitoring subsystem, invoke-rc.d smartmontools start.

There are also some very good answers at Monitor disk health using smartd (in smartmontools) on a high availability software RAID 1 server

Improve this answer
2

Because you can't foresee when and where a disk is corrupted, the easiest way to prevent that your backup is overwritten by a corrupted copy is doing a rotating backup.

So basically you can do daily backups to different locations. When you notice a disk fault and you have several backups to restore from, even if the last one is overwritten by a corrupted one.

Should be an easy task with cron and rysnc and there are several scripts for that purpose out there.

Improve this answer
3
  • I have this, but if a file is modified today and get corrupted, if I notice it two days after, restore the today's backup is not an option
    – slacker
    Commented Apr 28, 2015 at 12:03
  • As said you can't predict what and where a disk or file is corrupted. So you can't just monitor some test file, because it may be still ok... you have to monitor the files you don't want to get corrupted. Is there a chance to open the files more often by the application to check if they are still ok, so that faults are noticed earlier?
    – FloHimself
    Commented Apr 28, 2015 at 12:11
  • There are too many files per day, and none of them can be lost. Only the application writes uses too much I/O...check each of the million files every time would stop the application.
    – slacker
    Commented Apr 28, 2015 at 13:07
1

This sort of safety should not be tried to implement from scratch on top of a file system. Instead I suggest to reorganize your system and use ZFS. With ZFS any consistency is handled on the file system level without the need for you to keep track of checksums or other means, and without need to explicitly veryfy file state on every access or with every tool accessing files.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .