1

I have multiple directories named by date (ex: 2017-09-05) and inside those directories multiple log.gz files from BRO IDS. I am trying to enter each directory, and get only specific log.gz files by name, and send those to a remote system using rsync. A log file looks like this:dns.00:00:00-01:00:00.log.gz I am attempting to use wildcards to accomplish this.

Ex: rsync -avh --ignore-existing -e ssh -r /home/data/logs/2017-09-* {dns,http}.*.log.gz / root@10.23.xx.xx:/home/pnlogs/

This is close, but its just copying all the files in each folder and ignoring my attempt at getting just http and dns logs as seen in the example. Is this possible to do in one line? Is there a better method?

Ideally I would like to keep the files in their original directories upon transfer. For example in my original command the directories will stay on the remote system. This would be nice for organization's sake. Interestingly, that command copied HTTP and DNS logs from 2017-09-01 and 2017-09-02, but not from the 3rd, 4th, or 5th of this month.

How could I adjust this to account for changes in month/year? This will sit in a script and I wouldn't want to have to change it every month (i.e. to 2017-10, etc) just using $date?

Got this working with: rsync -avhR $(date +????-??-??)/{dns,http,conn}.*.log.gz root@10.23.xx.xx:/home/pnlogs/ Thanks for all the help.

  • Make a list of files, then use option --files-from of rsync. – Satō Katsura Sep 5 '17 at 18:04
  • 1
    Use $( ... ) for the date command interpolation. Backticks are deprecated these days. Also, why not just ????-??-?? instead of specifying a date. This way it will pick up older files that haven't been transferred, for example if the job fails to run one day. – roaima Sep 5 '17 at 19:34
2

Let's take a look at your command

rsync -avh --ignore-existing -e ssh -r /home/data/logs/2017-09-*  {dns,http}.*.log.gz / root@10.23.xx.xx:/home/pnlogs/

The collection source for this is all files (recursion is enabled with -r - but actually -a already does that) that match these four paths:

  • /home/data/logs/2017-09-*
  • dns.*.log.gz
  • http.*.log.gz
  • /

The last of these will ensure that the rsync command tries to copy everything.

I think what you're wanting to copy is all the DNS and HTTP log files inside the YYYY-MM-DD folders, keeping the directory structure:

cd /home/data/logs &&
rsync -avhR 2017-09-*/{dns,http}.*.log.gz root@10.23.xx.xx:/home/pnlogs/

I wouldn't use --ignore-existing unless you have a really strong reason for knowing you want to do that. (This flag prevents rsync restarting a partial transfer. Mind you, you don't have --partial so perhaps this is moot.)

You don't need -r because you're using -a and this archive mode already includes recursion. But if I've interpreted your requirement correctly there is no recursion anyway.

The -R flag tells rsync to keep the relative paths from the source trees in the destination. I've used cd to get the command into a useful starting point because otherwise you'd end up with /home/data/logs also being included in the target path. Wrap all of this in brackets ( ... ) if you want to avoid the directory change being effective for the remainder of any script in which this runs.

  • For the record, rsync uses a replace-by-rename model by default, so without --partial, you can resume an interrupted transfer just fine, provided you don't mind files that were already transferred but have changed on the source not being updated). From a practical perspective, --ignore-existing also makes things insanely fast for an initial transfer because it skips a lot of the expensive checks done for deciding what to transfer. – Austin Hemmelgarn Sep 5 '17 at 18:12
  • Is it possible to keep the files in their originals directories upon transfer? For example in my original command the directories will stay on the remote system. This would be nice for organizations sake. Interestingly, that command copied HTTP and DNS logs from 2017-09-01 and 2017-09-02, but not from the 3rd, 4th, or 5th of this month. – Blitzkrieg Sep 5 '17 at 18:15
  • @AustinHemmelgarn the --partial flag is not default. Here's what the man page has to say on the subject: « --partial By default, rsync will delete any partially transferred file if the transfer is interrupted. » – roaima Sep 5 '17 at 18:19
  • @roaima I never said that --partial was the default, I said that rsync uses a replace-by-rename method by default, which is controlled by --inplace, not --partial. Apologies if the wording of my comment implied that --partial was the default. – Austin Hemmelgarn Sep 5 '17 at 18:25
  • @Blitzkrieg if it's missing files then they didn't match the template pattern (2017-09-*/{dns,http}.*.log.gz) or else they were created after the rsync was begun. – roaima Sep 5 '17 at 18:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.