3

given a harddisk (old magnetic harddrive), with freshly synced data (backup / copy of other source), a lot of files are untouched (read but no write). Any harddrive looses data over time (you see bitflips, that is why you should have at least 3 copies on different hardware...).

So after syncing, I know, all data is in a correct state. Is there a way with standard shell commands* to read each file and rewrite (overwrite so the filesystem does not fragmentize) it? the intended purpose is to refresh the magnetic state of all bits. Or is any Linux filesystem capable of doing something like this [currently I prefer ext4]?

Or is there any argument that this idea is void?

*or any other clever tool

Improve this question
5
  • 1
    Why not simply store the data on a ZFS filesystem and periodically do a zpool scrub ...?
    – Jim L.
    Sep 15, 2021 at 21:24
  • @JimL.1st, ZFS is that with that huge RAM cache right? Is it really a good option for long term storage? 2nd: "The scrub examines all data in the specified pools to verify that it checksums correctly" sound like a read only to me... so it might fix a single or double bit-flip with its checksums, but the data is not refreshed really, so the likehood of bitflips will grow over time. 3rd: how much more space the redundant data needs? might be a lot... assume at least 1/3rd or so, right?
    – schnedan
    Sep 15, 2021 at 21:41
  • Bonusquestion for ZFS: is this a good choice with linux?
    – schnedan
    Sep 15, 2021 at 21:51
  • I'd suggest you read up on ZFS. ZFS is a self-healing filesystem, and can recover from any number of bit flips, so long as the redundant device(s) in the pool don't have any bit flips in the same block. No, ZFS doesn't need huge amounts of RAM, per se, especially if the ZFS pool is a long-term backup store. Typically one would use at least a mirrored pair of drives or better a RAIDZ-2 pool. Using the ZFS copies property to store multiple data copies on a single disk is not recommended.
    – Jim L.
    Sep 15, 2021 at 22:17
  • Note that ZFS detects such errors whenever a data block is read, not just when zpool scrub is run. Periodically running a scrub is just a way of ensuring that all used blocks are regularly read. Detected errors are automatically corrected if there is sufficient redundancy (i.e. a mirror or raidz vdev, or even via the "copies" property). BTW, I say "block" here rather than "file" because ZFS checksums are per-block, not per-file (and a file may have 0, 1, or many blocks) and also because ZFS supports zvols (which are analogous to LVM's logical volumes) as well as filesystem datasets.
    – cas
    Sep 16, 2021 at 9:34

2 Answers 2

Reset to default
3

The simplest and fastest way would probably be,

dd if=/dev/sda of=/dev/sda bs=256M

dd reads before writing (of course, or it wouldn't have anything to write), and device position is independent on source and destination.

Also, this assumes that the disk doesn't die on you during the refresh.

I'd run a speed test using the same overall amount of data, but a bs equal to one third, one half, exactly and twice the size of the hard disk cache. Depending on the hard disk controller's cache allocation strategy you might experience significantly different rewrite times.

So, for example, assuming a 256M cache,

time dd ... bs=80M count=1638 time dd ... bs=128M count=1024 time dd ... bs=256M count=512 time dd ... bs=512M count=256

However, bs should not exceed one full physical cylinder's worth of data, otherwise the mechanics of the hard disk will be forced to skip one track to read the data beyond the first cylinder, and then skip back to write the same data. You will easily recognize the situation because the hard disk will begin to stridulate. You want it to emit instead a determined "tic. tic. tic.", each cylinder being addressed only once. Determining physical size is difficult due to LBA mapping, and it might even be impossible on newer hard disks due to different cylinder sizes in the various allocation zones (at different distances from the spindle). In such cases, "refreshing" a disk will impart a significant stress on the drive mechanics.

Improve this answer
2
  • I like that solution, as it is low level and independent of the file system.
    – schnedan
    Sep 15, 2021 at 22:02
  • 1
    While this might "refresh" the data, it wouldn't detect or correct bitflips or other data corruption. It would simply copy the bad data.
    – mikem
    Sep 16, 2021 at 21:13
2

ZFS is the answer to most real-world on-disk data integrity issues. It will detect and correct bitflips, and any number of other data corruption events.

It's pretty stable on linux and rock solid on Solaris and BSD variants. If you don't trust it on Linux just yet, implement it on a file server somewhere and serve iSCSI, NFS, or CIFS volumes to the servers that need the data. ZFS on the backend will keep everything nice and safe.

Improve this answer
4
  • Well, just to clarify, my intend is not to run a server - its home-use, My self build NAS is based on OMV with a linux sw raid (zfs still not officially supported), serving me since years without problems and my question is based on a single hd which is stored in a safe place (fire/water) as emergency backup. This disc I want to refresh.
    – schnedan
    Sep 16, 2021 at 6:56
  • 1
    Use FreeNAS instead. It's BSD based and has native ZFS support. If you really want to stick with linux, I'd still recommend ZFS on linux. Simply put, if you want to ensure data integrity, ZFS is the answer. dd or any other copy mechanism won't identify corrupted data. ZFS checksums every block... if there's data corruption, it will find it, and tell you exactly which file(s) are corrupted. If the data gets corrupted in transit -- ie from memory to the controller to the disk, it will detect it.
    – mikem
    Sep 16, 2021 at 7:06
  • when I setup that particular NAS, FreeNAS was the first thing I tested, but it failed to boot. OMV worked right away. thats the linux advantage... it just supports more hardware
    – schnedan
    Sep 16, 2021 at 10:12
  • Yeah, I get that. Give ZFS on linux a try. I don't think you'll be disappointed.
    – mikem
    Sep 16, 2021 at 20:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .