rsync does not create destination folder as I want

Question

I have the following text file

...
/a/b/c/f1
/a/b/c/f2
...
/a/b/c/d1
...

where f1 and f2 are files, while d1 is a directory. I would like to copy the exact hiearchy in my destination folder (/z).

With bash, for each line lr readed from the text file, my script runs (I omitted the options for the sake of simplicity)

rsync $lr/ /z/$lr

The folder is copied correctly because I obtain /z/a/b/c/d1. Unfortunately, for the files results

/z/a/b/c/f1/f1
/z/a/b/c/f2/f2

How could I solve this problem using the same above-mentioned rsync invokation for both the files and directories?

Answer 1

Don't rsync the files manually. Use the --files-from= option to include your text file.

Rules for directories:

• Only the directory is created if the path doesn't end with a trailing /
• To sync a dir's contents, add a trailing / or option --recursive / -r alternatively

If you use option -a and want recursion, you need to add -r additionally. Make sure the paths in the file are relative to the source dir (with or without leading /).

A minimal example for your input would be:

rsync --files-from=textfile / /z

If you need to modify/filter the text file beforehand, you can pass it from stdin to rsync, e.g.

your_command textfile | rsync --files-from=- / /z

Answer 2

Let's look at your rsync command. You've omitted the options - which are important - so I've assumed you're using either -a (--archive) or -rt (--recursive --times), and I'll add -v (--verbose) for convenience too.

You're reading source files from this text file:

/a/b/c/f1
/a/b/c/f2
...
/a/b/c/d1

And you have this template, where each line from the text file is read into $lr:

rsync $lr/ /z/$lr

Let's apply an equivalent of lines f1 and d1 to your templated command and see what we have:

mkdir -p /tmp/733765
cd /tmp/733765
mkdir -p a/b/c/d1
touch a/b/c/{f1,f2,d1/f3} z

rsync -av a/b/c/f1/ z/a/b/c/f1
rsync -av a/b/c/d1/ z/a/b/c/d1

Straightaway we can see that with the information shown to us, the file f1 will not get copied. Instead we get this error,

rsync: [sender] change_dir "/tmp/733765/a/b/c/f1" failed: Not a directory (20)

If you see f1 in the destination it must have been copied using some other solution, perhaps an earlier attempt at using rsync.

Similarly, d1 won't get copied to the destination unless you have already created the directory path z/a/b/c:

rsync: [Receiver] mkdir "/tmp/733765/z/a/b/c/d1" failed: No such file or directory (2)

mkdir -p z/a/b/c
rsync -av a/b/c/d1/ z/a/b/c/d1

created directory z/a/b/c/d1
./
f3

sent 122 bytes  received 71 bytes  128.67 bytes/sec

There's something not right about the information in your question. If we ignore the rsync command and concentrate on the source data set, you can use rsync to read the source data set directly:

rsync --files-from /path/to/source_data_file -arv . z/

This may seem slightly strange because usually -a includes -r. However because we are using --files-from and you have directories specified in the source data file we need to specify -r explicitly to tell rsync to recurse into the listed source directories. (Without this you'll only get named files copied.)