1

I have a ZFS system connected to a macintosh over NFS. The system is nearly full, but I have found a large number of duplicate files with jdupes.

I don't know how these copies were made, it's possible they were redownloaded from the internet, and it's possible they were copied with cp. If they were copied locally, they should not be using any additional space, as ZFS is a copy-on-write filesystem, but if they were redownloaded, they should be using up the same space twice. My NFS server doesn't have enough RAM for ZFS deduplication.

I'd like to replace any physically duplicated copies with copy-on-write copies.

How can I tell which file duplicates are copy-on-write duplicates of each other?

1 Answer 1

1

It seems that you are confusing two properties of ZFS here:

  • Copy-on-Write (CoW) is a description of how the file system works when modifying files. Traditional (non-CoW) file systems have a read-modify-write cycle, where updates to a file are done in-place, meaning the only copy of the file is modified directly (writing the updated bytes on the specified position inside the file). If power is lost, damage may occur. This is not the case on CoW file systems like ZFS, where any modification to a file does never touch the file: the changed bytes are simply appended and if that is successful, the old parts are marked as deletable. This prevents loss of integrity at the cost of more fragmentation (because each change needs a new different space).
  • Deduplication is an optional feature that can be enabled on ZFS file systems (default is disabled). If enabled, any data to be written is first looked up on the deduplication table (DDT). If it already exists, it is not written again, only linked. This works in transit (reading and writing), but not for already duplicated stored "old" data. It also has certain downsides that you can read all over the internet, so I will not outline them here.

If you have never enabled deduplication on your file system, any copy (cp, copy in Finder etc.) will create new data and reduce your usable space. There is no built-in default deduplication (even the recommended compression is disabled by default!). You can check this for yourself by copying a file of 1 MB 1000 times, you will lose 1000 MB of space instead of 1 MB.

How can I tell which file duplicates are copy-on-write duplicates of each other?

You can't, but you could use a duplicate checker like you did to find exact duplicates (like on any other file system). You could also use zdb -b poolname to estimate the benefits of deduplication to see if RAM increase would be good for your case. Also, enabling compression will help in any case, so if you do not already use it, enable it.

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.