4

I have a remote server with an rsync share and I need to pull just the most recent 7 days of files.

I can SEND files based on date by using:

find /path -mtime 7 ... -exec rsync -a

but I can't just login to the server and send the files to my local machine so I need a way to use rsync to get all the files that are less than 7 days old and grab them.

I cannot simply sync everything as the total data on the server exceeds 100TB and I don't have access to the remote server other than the rsync module.

  • Wouldn't that run find locally? Or am I missing something? – lbutlr Aug 5 '16 at 5:34
  • Can you get a file listing including time stamps from the remote side? (If all else fails, you could touch dummy files locally and use --ignore-existing to not overwrite those files.) – Ulrich Schwarz Aug 5 '16 at 6:33
  • I assume you have ssh-access when using rsync, so you could use ssh backup@host "find <dir> <options>" > temp_file_list and then use this list for rsync. – Fiximan Aug 5 '16 at 7:54
6

Surprisingly, this can actually be done with rsync(1) alone, without shell access to the remote server.

Assuming bash(1) and GNU date(1):

#! /usr/bin/env bash

src=rsync://example.com/archives/
dst=/path/to/mirrors

cutoff=$( date -d '7 days ago' +%s )
rsync -na --no-motd --out-format='%M %f' "$src" "$dst" | \
    while IFS= read -r line; do
        d=${line%% *}
        fn=${line#* }
        fdate=$( date -d "${d/-/ }" +%s ) || continue
        if [ $fdate -ge $cutoff ]; then printf '%s\0' "$fn"; fi
    done | \
    rsync -a --files-from=- -0 "$src" "$dst"
  • I am getting file_stat errors and I can't figure out why. Given src=rsync://server/files/4K/; dst=/local/4K/ I get errors like rsync: link_stat "/4K/local/4K/folder one/File Name 4" (in files) failed: No such file or directory (2) I mean, I understand the error Is because that path is invalid, but I don't understand how that path is getting mushed together like that. – lbutlr Aug 8 '16 at 23:09
  • @lbutlr As far as I can tell that's because of the trailing slash in dst. rsync does some bizarre dance with parsing trailing slashes. To mirror directory /files/4K on the server to local directory /local/4K you need src=rsync://server/files/4K/ (with trailing slash) and dst=/local/4K (without trailing slash). That's probably all you need to remember, but there's a lot more about it in the manual. – Satō Katsura Aug 9 '16 at 7:44
  • Hmm. Took out the trailing slash and it looks the same. I wonder if the source server is using an old rsync. Does this require rsync 3.x? – lbutlr Aug 9 '16 at 9:37
  • @lbutlr No idea. But your problem is No such file or directory (2), not protocol mismatch, option unknown, or similar. Perhaps a dangling symlink on the server? – Satō Katsura Aug 9 '16 at 9:39
  • The end of the path on the server "4K" is getting prepended to the path of the local share (/local/4K) and THAT is being passed to rsync along with the file path, which cannot find the file on the server, of course. – lbutlr Aug 11 '16 at 16:27
4

This is the approach that I used as part of an rsnapshot backup script:

rhost=user@backuphost
rpath=/path/to/files

ssh -qx "$rhost" "cd '$rpath' && find . -mtime -7 -print0" |
    rsync --from0 --files-from=- -avHS "$rhost:$rpath/" .

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.