1

I have a semi-working rsynccommand that I have combined with the find command on a linux box to do a simple file transfer of a specific type and date.

Here's the command:

rsync -avx --timeout=30 --ignore-existing admin@host:'`find /results/analysis/ -mtime -1 -type f -iname  "*.xq"`' /home/serverdir/

Here's the problem:

The above command works perfectly if any files are found. It breaks when no files are found and actually sends a file that is located on the home directory of the remote machine for some reason, which by the way is much older than 1 day as indicated by -mtime.

The error message:

receiving incremental file list
./
rsync: send_files failed to open "/home/admin/.viminfo": Permission denied (13)
file.txt

It's as if when the find command reports nothing, rsync just defaults to sending everything in the home directory. Any ideas of how to fix this?

1

Rather than mixing find and rsync, since all you appear to want to copy are the *.xq files, you can tell rsync to do this directly. I've included the --dry-run flag so that you can safely test it without any changes being effected; when you're ready simply remove it from the command line.

rsync --dry-run -av --include '**/' --include '*.xq' --exclude '*' admin@host:/results/analysis/ /home/serverdir/

The mixture of --include and --exclude is explained as follows, in order from left to right:

  1. --include '**/' include all directories (but not files)
  2. --include '*.xq' include all files (or directories) that match the glob *.xq
  3. --exclude '*' exclude everything not already mentioned
| improve this answer | |
  • in addition to the file type, I need to specify only files that have been modified in the past 24 hours. Interestingly enough though, I added the --exclude '*' flag you mentioned above and the script seems to work even when there are no files present that have been modified in the past 24 hours. So the new command looks like this: `'rsync -avx --timeout=30 --ignore-existing --exclude '' user@host:'find /results/analysis/ -mtime -1 -type f -iname ".xq" /home/serverdir/' – Sam Mar 12 '15 at 0:24
  • @Sam why do you need to include only files modified in the past 24 hours? Can't you just let rsync skip them with no changes? – roaima Mar 12 '15 at 0:25
  • 1
    I can't have those files being transferred to the destination. I have a directory full of thousands of old files and only the files from the past 24 hours are to be used on the local machine. Basically I have a requirement to not only put new files on the local machine but also keep out the old ones as well. – Sam Mar 12 '15 at 0:32
1

What you observe is exactly what is happening: just HOST: without an explicit path asks to copy the entire default directory (usually the home directory):

Note  also  that  host  and  module references don’t require a trailing
slash to copy the contents of the default directory.  For example, both
of these copy the remote directory’s contents into "/dest":

       rsync -av host: /dest
       rsync -av host::module /dest

I would recommend either avoiding the use of find entirely, if that is possible (remember that rsync will use negligible bandwidth for already-transferred files), or doing the deed in two steps (first find, then rsync over its output).

More precisely, something like the following should work (I omitted some of your rsync options for brevity):

ssh user@host find source-dir -mtime -1 -type f -printf %p\\\\0 \
| rsync --from0 --files-from=- user@host:. dest-dir/

The main change from what you mentioned as a comment: the find command runs on the remote host rather than locally.

| improve this answer | |
  • what would be the syntax to run find first using a bash terminal? I tried something like: find user@host:/folder/ -mtime -1 -type f -name "*.xq" -printf %P\\0 | rsync -avx --timeout=30 --files-from=- --from0 /folder/ /local/folder/ But that doesn't work because evidently find can't be called like that. – Sam Mar 11 '15 at 22:44
  • What I had in mind is very similar to what you wrote here; I will update my answer with something that is likely to work. – dhag Mar 12 '15 at 4:03
  • Thanks @dhag, also I have something that works based on the comment below: {rsync -avx --timeout=30 --ignore-existing --exclude '' user@host:'find /results/analysis/ -mtime -1 -type f -name "*.xq"' /local/folder/} the key was for me to use the --exclude "" option. – Sam Mar 12 '15 at 17:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.