5

So in this story on the GNU coreutils ML, someone used cp to copy 430 million files and needed to preserve hard links, and just barely got it to finish after 10 days.

The big problem was that, in order to preserve hard links, cp has to keep a hashtable of already copied files, which took up 17GB of memory towards the end and had the system thrashing on swap.

Is there some utility that would have handled the task better?

7
  • rsync has done very much optimizations, so it might be a good option to try. Perhaps cpio could work too. I think the trick must be to do it in portions, perhaps even in parallel, if you want max speed.
    – MattBianco
    Sep 12 '14 at 10:34
  • 1
    @MattBianco: the problem is that you can't preserve hard links if you do it in portions unless you repeat a lot of work. Sep 12 '14 at 10:40
  • Based on reading that thread, it seems they have made some fixes to the code. Does that not help? Have you tested the cp with the new fixes? Sep 12 '14 at 10:45
  • Hrrm. Maybe the tar trick? I don't know if/how that handles hardlinks, though.
    – evilsoup
    Sep 12 '14 at 11:20
  • 1
    Files with only 1 link could be done in portions. Finding those is cheap. Are the majority of the files "duplicates"? I think the best approach could be different depending on circumstances like the structure of the data, as well as of course which filesystem the files reside on.
    – MattBianco
    Sep 12 '14 at 11:21
1

If the tar or rsync solutions fails and if the directory is the root of a filesystem you can use the old dump/restore backup utilities (yes that stills works).

dump duplicates the filesystem characteristics without going through the kernel filesystem interface so it is quite fast.

The inconvenient is that dump is sensible to modifications made on the source file system while copying. So better umount the filesystem or remount it read only or stop any application that could access it before starting a copy. If you respect that condition the copy is reliable.

Depending on the filesystem type the dump command name can change, for instance, you can have the xfsdump for the XFS.

The following command is similar to the tar example :

dump 0uf - /dev/sdaX  | (cd /target && restore rf -)

The number is the incremental copy level; 0 indicates to do a full copy.

2
  • Let's just add, that dump is extX only, but similar utilities exist for other filesystems.
    – peterph
    Sep 12 '14 at 15:03
  • @peterph Yes you are right that depends on the operating system too.
    – Emmanuel
    Sep 12 '14 at 17:03
0

I recall rsync resulted in some problems for me; I am not sure if these were hard links or device files or whatever. Since then I use

tar -cv $(ls) | ( cd /mnt; tar xv )

which will copy all files from . to /mnt. My whole story can be found here: http://www.linuxintro.org/wiki/Cloning#file-by-file_copy_2

3
  • Is this able to handle hardlinks?
    – evilsoup
    Sep 12 '14 at 13:39
  • @evilsoup see my comment on the question, the one beginning with cpio. Gnu tar can recreate hard links somehow, but I doubt that it can do it effectively on the kind of volumes mentioned in the question. I guess someone has to try it...
    – MattBianco
    Sep 12 '14 at 13:57
  • GNU tar without -f defaults to stdin/stdout. Others may not do so. Some default to tape devices. I recommend using tar cf - $(ls) | (cd /mnt && tar xf -). Replacing ; in the subshell with && protects you from cd typos. Skipping v saves you from tars that are verbose on stdout.
    – MattBianco
    Sep 12 '14 at 14:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.