1

I'd like to copy from /A to /C only paths missing on /B and /C. (Please assume those are paths, and can represent also remote locations e.g. mounted via sshfs)

I wonder... Is there more concise way then writing a loop, iterating over filesystem and making check, file by file ?

Example contents of tree directories /A, /B, /C :

/A:

/A/1abc/qwe
/A/2abc/asd
/A/3abc/zxc
/A/4abc/rty

/B:

/A/2abc/asd
/A/3abc/zxc

/C:

/C/1abc/qwe

expected: to copy from /A to /C only path 4abc/rty:

/A/4abc/rty -> /C/4abc/rty

To illustrate practical examples of /A, /B, /C, leading to such scenario:

  • you make backup, copy from some /source (/A) to some /destination (/B) and some paths failed to be copied. So you want to have copy anyway of missing ones. So you find /another_destination (/C) that can accept them, and you want to copy only missing ones. Here are example limitations why copy from /A to /B could fail: disc got full; limitation of filesystem of /B (like filename lenght), while not present on /C filesystem; etc.
  • A,B & C are on same machine or are they remotely resided ? – Rahul Jun 20 '16 at 10:35
  • Please assume that all are mounted, but might be over sshfs (so remotely over not fastest connection). – Grzegorz Wierzowiecki Jun 20 '16 at 11:04
  • your example is bit confusing, regardless of your title. If you can edit more it would be nice for me or for others too. – Rahul Jun 20 '16 at 11:26
  • Sure! Let me rewrite a bit once I am back at desktop :). – Grzegorz Wierzowiecki Jun 20 '16 at 11:29
1

One solution is to make a union mount of B and C with C as the write branch. Then the problem reduces to copying from A to B+C only files missing on B+C, which can be done with

rsync -a --ignore-existing /A /B+C

For the union mount, one possibility is unionfs-fuse.

unionfs-fuse -o cow /C=RW:/B=RO /B+C

Or unionfs:

mount -t unionfs -o dirs=/C=rw:/B=ro unionfs /B+C 
  • By curiosity : why unionfs-fuse instead of in-krenel unionfs ? I guess, it was just hany to write for you but both would do the job - or are there some edge cases where uninfs-fuse would be better for this kind of application ? – Grzegorz Wierzowiecki Jun 21 '16 at 12:49
  • 1
    @GrzegorzWierzowiecki It has just handy for me to write, most union mount systems can handle this. For a discussion of union mount systems, see this thread — when somebody answers! – Gilles Jun 21 '16 at 12:56
  • Too many open files (24) by rsync. I wonder if setting higher like -o max_files=32000 option for unionfs-fuse is good strategy? – Grzegorz Wierzowiecki Jun 26 '16 at 21:31
  • No, it doesn't help. Still I get rsync: chown "/.../xyz" failed: Too many open files (24) at some point and rsync does not rsync everything, and unionfs-fuse directory is not possible to ls anymore. Maybe I should try with unionfs or AuFS instead. I couldn't find rsync flag that could help for that. – Grzegorz Wierzowiecki Jun 27 '16 at 14:05
  • @GrzegorzWierzowiecki Oh, I'd never encountered that problem with unionfs-fuse. Presuming that unionfs-fuse is the culprit (the open file limit is per-user), another union mount implementation might help, and an in-kernel one (overlayfs, aufs) would not run into this problem. – Gilles Jun 27 '16 at 18:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.