I have a large set of files and need to upload them to a remote machine with specific destinations. I have a map (can be re-organised) as

1/1/1/file.jpg -> 2/3/4/image1.jpg
2/12/2/file.jpg -> 5/6/7/image2.jpg
3/31/31/file.jpg -> 8/9/1/image3.jpg

I can think of two solutions:

Uploading to specific destinations

scp /folder/1/1/1/file.jpg user@ip:/dir/2/3/4/image1.jpg
scp /folder/2/12/2/file.jpg user@ip:/dir/5/6/7/image2.jpg
scp /folder/3/31/31/file.jpg user@ip:/dir/8/9/1/image2.jpg

but it is not efficient to have thousands of scp connections.

Question: How can I upload multiple files in one connection (for the sake of speed) with a method like scp?

Uploading with rsync and then batch rename

I can upload the entire folder with rsync, which is quite fast, to a temporary folder. Then, rename the files according to the above map.

Question: How can I batch rename with a given map?


A third solution requires that all of the source files are on the same filesystem, and thus hard links can be made arbitrarily to any of the files.

Store your file map in file filemap.txt:

$ cat << EOF > filemap.txt
1/1/1/file.jpg -> 2/3/4/image1.jpg
2/12/2/file.jpg -> 5/6/7/image2.jpg
3/31/31/file.jpg -> 8/9/1/image3.jpg

Then use this script to create a temporary output directory named temp which will contain sub-directories matching the destination paths of your file map. Each source filename will in turn be hard-linked into its proper destination path under directory temp. Once temp is fully populated, it will be rsynced to user:ip and then deleted from the local machine.

#!/usr/bin/env bash


rm   -rf temp

while read  src X dest

        dir="$(dirname "$dest")"

        mkdir -p "temp/$dir"
        ln "$srcD$src" "temp/$dest"

done < filemap.txt

rsync -av temp/ user@ip:"$destD"

rm -rf temp
  • brilliant solution indeed, because I have to create the map file. With this idea, I directly create symbolic links instead of the map file.
    – Googlebot
    Apr 23 '21 at 22:51
  • your rsync command needs L.
    – Googlebot
    Apr 23 '21 at 23:30
  • @Googlebot Probably because you're using symbolic links, whereas my example uses hard links.
    – Jim L.
    Apr 24 '21 at 6:06
  • you're absolutely right. Is there any benefit to using a hard link here? I gather a soft link might be more convenient as the purpose is temporary.
    – Googlebot
    Apr 24 '21 at 11:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.