1

Is there a command line tool that can be used to rewrite all regular files in a directory tree either in-place or by creating new inodes?

With rewriting a file in-place, I mean opening the file for reading and writing, reading blocks of a reasonable size and writing those blocks at the same location, doing this for the whole file. Basically what this command line does:

find dir -type f -print0 | xargs -0 -n1 bash -c 'dd if="$1" of="$1" conv=notrunc bs=64M' -

If instead a new inode is created, file attributes should be replaced as good as possible, e.g. what this command does:

find dir -type f -print0 | xargs -0 -n1 bash -c 'echo "$1"; cp -a "$1" "$1~" && mv "$1~" "$1"' -


Background:

I'm in the process of trying to gain some experience and finding good practices for using ZFS deduplication, where appropriate. ZFS deduplication uses a DDT (deduplication table) and operates on blocks of a size given by the file's recordsize, which has an impact on the effectiveness and memory-usage of deduplication. I'm exploring the possibilities of migrating already-written data to use or stop using the DDT or change the file's recordsize. ZFS does not automatically change these parameters of already-written data, so the data needs to be rewritten.

To change whether the DDT is used, it is sufficient to rewrite the data in place (without creating a new file). But the recordsize of a file is determined when it is created and thus a new file needs to be created to change it.

8
  • I'm virtually certain there's no such tool. It's such a specific, unusual action that I can't imagine why anyone would create it. – Barmar Mar 4 '20 at 17:17
  • I would have guessed there was one e.g. for people who want to defragment their files after downloading them using BitTorent to a lazily allocated file on a non-copy-on-write file system. ¯\_(ツ)_/¯ – Feuermurmel Mar 4 '20 at 17:57
  • I think most filesystems will simply overwrite the existing block, not allocate new blocks, when you write a file in place, so it won't defragment. – Barmar Mar 4 '20 at 18:00
  • Oh, I wasn't specific enough, I was thinking about a tool implementing the second variant for that use case. – Feuermurmel Mar 4 '20 at 18:01
  • 1
    @Feuermurmel Many torrent clients (e.g. deluge) have an option to move a file after the download has completed. This effectively de-frags it if it is moved to a different drive or partition (you can use, e.g., a loopback mount for this if your system only has one filesystem partition). For ZFS, it is recommended to download to one dataset (for deluge, at least, it's also recommended to use a 16K recordsize for that dataset) and move it to another on completion. Finally, it's also a good idea to disable any automatic (e.g. via cron) snapshotting on the temp download dataset. – cas Mar 28 at 6:04
2

I just created tool that does just that:

https://github.com/pjd/filerewrite

Alternatively with ZFS you can also use zfs send/recv where the target file system has deduplication turned on. After that you will need to rename the file systems and make sure all the other file system properties are moved over.

1
  • Haha, you're good! Your tool looks legit. :D I'm gonna test-drive it when my use-case comes up again. – Feuermurmel Mar 31 at 11:53
1

(This question is a year old as I write this, but it came up in my review queue, so I'll answer it in case anyone else searches for a similar need.)

It is very unlikely such a tool exists and is reliable across different filesystems types, especially in a *nix system that may have multiple filesystem types mounted at various points of the system root tree. The reasons for this are that some filesystems may do additional work "under the hood", and cannot guarantee that a rewrite will occur in the same place, (In effect, what you are wanting to do--get ZFS to do exactly that with de-duplication by re-writing the file data as it sees fit!) unless the filesystem already supports such a function itself.

You could write a quick program to open a file in read+write mode, then read a block, seek back to the start of the block, and write it again. I do not know if this would work with ZFS de-duplication, or what other kinds of issues you may encounter. Encrypted filesystems, for example, may need to re-encrypt the file, and/or even move where it is physically stored on the physical storage device. This is a part of why you'd need to write such a tool yourself to your specific needs.

Another approach would be to make a backup of the file tree, then restore the backup over the original. As noted, however, this requires you have a backup storage of some sort (although it can be a simple tar file elsewhere on the same physical disk, etc.), and choose your backup and restore options appropriately.

As a last resort, you might be able to rename the directory tree elsewhere, then re-create it by copying one file at a time (deleting the old after you verify the new is safely copied, to free up space). You will still need enough extra storage for making the copy of largest single file. This is not easily avoidable in the majority of scenarios, especially if the filesystem driver tries to be efficient enough to realize the data has not actually changed (keeping a block hash/checksum, for example) and simply discard the unnecessary write. Memory-to/from-memory operations are significantly faster for a computer than memory-to/from-device operations.

2
  • Hi! Thanks for the detailed answer! I agree that depending on the file system type what I want the file system to do, more elaborate approaches are necessary. For my use-cases limited to ZFS, the approaches I drafted in my questions (with either dd or cp) are sufficient (ZFS is a very deterministic file system in that respect). – Feuermurmel Mar 31 at 11:49
  • In such a case, I believe Pawel's tool will be good enough. I did not look closely at the source, but i seems to work as I described: read-seek-write, and resets the file modification time when done. – C. M. Mar 31 at 22:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.