I have gigantic data consisted of branching tree of folders, sub folders, sub-sub folders and sub-sub-sub folders. Inside the terminal sub folder, I have large number of images (~ 2000 image each file is ~ 5-- kib ) which makes opening the folders very hard and slow. The images are not traditional images. they have header contain information. This header is common between all the images. This data is located on an old server which has low capabilities.

I want to make copy of the tree to another server which has better performance and copy one file (anyfile) within each sub folder to the new server.

I used

rsync -av -f"+ */" -f"- *" /path/to/src ssh user@xx.xx.xx.xx:/path/to/dest/

This command, really, created a new empty tree oon the new server. My question is:

How can I revise this command to enable it sync one file/ any file of every folder?

my question is more focused on the roles defined by the flags f how can I add new flag f for new role that tells rsync to sync one file from each folder and sub folder?

I can't use any solution that include cp mv or find. Running these commands on the old server and the huge amount of data will take forever!

  • 4
    What does "first" mean in this context? The oldest? The one whose name is alphanumerically smaller than the rest? – Andy Dalton Oct 4 '18 at 19:23
  • Why do you want to copy only one file from each folder? Maybe I'm missing something in your explanation. – roaima Oct 4 '18 at 21:03
  • 1
    @Zahi. Are the filenames in a particular folder have pattern of naming convention? – user88036 Oct 4 '18 at 21:31
  • You do realise that although find/cp will traverse the filesystem, rsync will also traverse it? One is unlikely to be significantly slower than the other during traversal. – roaima Oct 5 '18 at 18:34

Do this after your rsync command

find /path/to/src -type d | while read D; do F=$(find ${D} -maxdepth 1 -type f | head -1); test -z ${F} || cp ${F} "/path/to/dest$(echo ${F#path/to/src})"; done

This will copy the first file from each dir that contains any files

  • @Zahi, even with several thousand directories, each having just a thousand or so small files, it isn't a large amount of data. Really. – roaima Oct 4 '18 at 21:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.