17

I have no experience with btrfs, but it's advertised to be able to de-duplicate files.

In my application, I'd need to duplicate whole directory trees.

From what I learned, btrfs only de-duplicates in some post scan, not immediately. Even just using cp doesn't seem to trigger any de-duplication (at least, df shows an increased disk usage in the size of the copied files).

Can I avoid moving data around altogether and tell btrfs directly to duplicate a file at another location, essentially just cloning its metadata?

In essence, similar to a hardlink, but with independent metadata (permissions, mod. times, ...).

7
  • 8
    cp --reflink=always.
    – mikeserv
    Dec 16, 2015 at 17:35
  • 4
    Note that this isn't anything like a hardlink. When you cp --reflink=always, the result from the user perspective will be two completely independent files in every way. The fact that the underlying file system is abstracting that via copy-on-write is only an implementation detail. You don't get "a hardlink, but with independent metadata.". To my knowledge, btrfs doesn't do any automatic deduplication yet. I think that's a future plan but I'm not positive on that.
    – ormaaj
    Dec 16, 2015 at 18:32
  • @ormaaj - a hardlink wouldn't have independent metadata. and Udo asked for an implementation detail. when you do a reflink to a file you essentially clone its metadata. its only when the references independently change that the files diverge - and that's what deduplication is all about!
    – mikeserv
    Dec 16, 2015 at 20:24
  • 1
    @mikeserv Er, I'm pretty sure deduplication has a different sense. Deduplication is taking already existing redundant copies of data and re-unifying it. COW is a means of minimizing duplication, it isn't deduplication.
    – ormaaj
    Dec 16, 2015 at 20:45
  • @ormaaj - i think thats a weird thing to say: deduplication is not about minimizing duplication.
    – mikeserv
    Dec 16, 2015 at 21:10

1 Answer 1

19

There are two options:

  1. cp --reflink=always
  2. cp --reflink=auto

The second is almost always preferable to the first. Using auto means it will fallback to doing a true copy if the file system doesn't support reflinking (for instance, ext4 or copying to an NFS share). With the first option, I'm pretty sure it will outright fail and stop copying.

If you are using this as part of a script that needs to be robust in the face of non-ideal conditions, auto will serve your better.

5
  • are you Eric Estrada?
    – mikeserv
    Dec 20, 2015 at 2:27
  • 6
    @mikeserv Lol, no. My first name is Ethan. That would be funny though; Eric Estrada: actor by day, sysadmin by night. Believe it or not, this is the first time in over a decade of going by the online handle eestrada that anyone has ever asked me that.
    – eestrada
    Dec 20, 2015 at 2:42
  • 5
    sure, Eric. anyway, good answer.
    – mikeserv
    Dec 20, 2015 at 5:38
  • If anyone is going to answer after 5 years is there any harm in doing this? alias cp='cp --reflink=auto' ?
    – Matt
    Mar 12, 2021 at 16:37
  • 2
    @Matt So long as you know your alias will always be run on a relatively recent version of the GNU userland, I don't personally see any harm in using alias cp='cp --reflink=auto'.
    – eestrada
    Mar 14, 2021 at 3:40

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .