3

I have a find command that finds certain files and directories. That find command then runs rsync with the files and directories found previously as source. The problem is that those files and directories can have all sort's of characters such as single and double quotes not to mention illegal characters from Windows etc...

How can I dynamically escape a string for use in rsync or other commands?

This command works by hard coding double quotes for rsync source string, but it will break if the string has double quotes in it.

find "/mnt/downloads/cache/" -depth -mindepth 1 \( \
-type f \! -exec fuser -s '{}' \; -o \
-type d \! -empty \) \
\( -exec echo rsync -i -dIWRpEAXogt --numeric-ids --inplace --dry-run '"{}"' "${POOL}" \; \)

resulting output:

rsync -i -dIWRpEAXogt --numeric-ids --inplace --dry-run "test/this " is an issue" /mnt/backing

Working command after info in answers applied:

find "/mnt/downloads/cache/" -depth -mindepth 1 \( \
                             -type f \! -exec fuser -s {} \; -o \
                             -type d \! -empty \) \
                             \( -exec rsync -i -dIWRpEAXogt --remove-source-files-- "${POOL} \; \) \
                             -o \( -type d -empty -exec rm -d {} \; \)
3

Your quoting problem is coming from you trying to solve a problem you don't have. Needing to quote arguments only comes into play when you're dealing with a shell, and if find is calling rsync directly, there is no shell involved. Using visual output isn't a good way to tell if it works or not because you can't see where each argument begins and ends.

Here's what I mean:

# touch "foo'\"bar"

# ls
foo'"bar

# find . -type f -exec stat {} \;
  File: ‘./foo'"bar’
  Size: 0           Blocks: 0          IO Block: 4096   regular empty file
Device: fd00h/64768d    Inode: 1659137     Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1004/ phemmer)   Gid: ( 1004/ phemmer)
Access: 2017-12-09 13:21:28.742597483 -0500
Modify: 2017-12-09 13:21:28.742597483 -0500
Change: 2017-12-09 13:21:28.742597483 -0500
 Birth: -

Notice that I didn't quote the {} in the arg to stat.

Now that said, your command is going to be very non-performant, because you're calling rsync for every single matching file. There are 2 ways you can solve this.

As others have indicated you can use pipe the file list to rsync on stdin:

# find . -type f -print0 | rsync --files-from=- -0 . dest/

# ls dest/
foo'"bar

This will use null bytes as the file name delimiter since files can't contain null bytes in their name.

 

If you're using GNU find, you have another method of invoking -exec, and that's -exec {} +. In this style find will pass more than one argument at a time. However all the arguments are added to the end of the command, not in the middle. You can address this by passing the arguments through a small shell:

# find . -type f -exec sh -c 'rsync "$@" dest/' {} +

# ls dest/
foo'"bar

This will pass the list of file to the sh which will then substitute them in for the "$@"

  • So how would you test if rsync is going to execute the correct comamnd then? I simply printed the actual command with -exec echo rsync ... Wouldn't that output the rsync command exactly as it would be executed? The problem with the rsync command I see is that it has no quotes and so will fail, example: rsync -args /src/no good /dest – DominicM Dec 9 '17 at 20:15
  • It depends on your definition of "as it would be executed". Yes it prints the command, but as I explained, you can't visually see the separation between arguments. You're not feeding the command through a shell, thus the shell definition of argument splitting, quoting, etc, does not apply. If your objective is to be able to know the exact command that would be run without running it, I'd suggest starting a new question, as there are several ways you could do it, and that's not in the scope of what was asked here. – Patrick Dec 9 '17 at 20:52
  • @DominicM, take the args script from mywiki.wooledge.org/WordSplitting and run that instead of echo, it'll show you the actual arguments more clearly (bracketed within <...> instead of just separated by whitespace) – ilkkachu Dec 10 '17 at 10:22
  • @Patrick, don't you need -exec sh -c '...' sh {} +, since the first argument after the -c script goes to $0, and that's not included in $@ ? – ilkkachu Dec 10 '17 at 10:25
  • @Patrick Took me a while but I get it now, and it works perfectly. Regarding pipe example, how would I apply another -type d and -exec command after the rsync pipe? I added my current code to my question to illustrate what I am trying to achieve. – DominicM Dec 10 '17 at 13:01
1

Use find's -print0 in combination with rsync --files-from=- -0:

find "${Share}" -depth -mindepth 1 \( \
-type f \! -exec fuser -s '{}' \; -o \
-type d \! -empty \) -print0 | rsync --files-from=- -0 "${Share}" ${POOL}

Now find will generate a list of found files separated by \0 (due to -print0). NUL isn't a valid character for a file name, so it's safe to use as a delimiter in a list. That list is printed to the standard output and read by rsync. rsync usually gets the source files as arguments, but can read a file instead:

find $PATH [your conditions here] -print > my-file-list.txt
rsync --files-from=my-file-list.txt $PATH $DESTINATION

Of course, since we used -print0 and not -print, we need to tell rsync that the files are delimited by \0:

find $PATH [your conditions here] -print0 > my-file-list.txt
rsync --files-from=my-file-list.txt -0 $PATH $DESTINATION

We can use - as argument for --files-from to tell rsync that the files should be read from its standard input and get rid of the intermediate file:

find $PATH [your conditions here] -print0 \
  | rsync --files-from=- -0 $PATH $DESTINATION

Note that rsync still needs a path as source, but it will only transfer the files found by find.

  • This is making my head hurt :) I see no reference in the manual to --from-file so I will need a much more in depth explanation to understand. Also is print0 needed when rsync executes exactly once per file/dir found by find command? I mean I dont see a need to delimit when there will only be 1 string for rsync to work with. Perhaps you could illustrate with the expected resulting rsync command output. – DominicM Dec 9 '17 at 17:56
  • @DominicM sorry, it's --files-from, no idea how I messed that up. And long story short, yes -print0 is needed, since you cannot use -exec with "weird" file names reliably, since -exec only replaces {}, as long as you use the ; variant. The + variant does not work since rsync expects the destination as last argument. If rsync used the destination as first argument, followed by the files you want to transfer, you could simply use -exec magicrsync DEST {}+. – Zeta Dec 9 '17 at 18:13
  • so why exactly is --files-from used here at all? I mean if "-" makes it use standard output, then why use this at all and not just use standard output in the first place? – DominicM Dec 9 '17 at 18:29
  • @DominicM because rsync does not use standard input by default? E.g. echo <filename> | rsync $PATH $DEST won't copy <filename>, echo <filename> | rsync --files-from=- $PATH $DEST will. – Zeta Dec 9 '17 at 19:21
0

Getting the quoting right in this context is devilishly hard. I usually give up and let other tools try it. Sorry this is untested, I don't have your input to try with, so this is a guess that could be wildly broken :) Use print0 on find to null-terminate (instead of blank-terminate) the outputs, and then -0 on xargs to tell it to expect that kind of input... this is to avoid breaking on embedded blanks.

find "${Share}" -depth -mindepth 1 \( \
-type f \! -exec fuser -s '{}' \; -o \
-type d \! -empty \) -print0 | \
xargs -0 -I{} rsync -i -dIWRpEAXogt --numeric-ids --inplace --dry-run {} ${POOL}

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.