4

I have a directory structure like so on a remote server, listing some of the files below:

/logs/service::A1/20210730T120000/log.log
/logs/service::A1/20210729T120000/log.log
/logs/service::A2/20210730T120000/log.log
/logs/service::B0/20210730T120000/log.log

In order to rsync all files from serviceA, I just run the following command with a single wildcard *

rsync -av <remote-server>:/logs/service::A* <destination>

This works as expected and I get the following directory structure at destination:

<destination>/service::A1/20210730T120000/log.log
<destination>/service::A1/20210729T120000/log.log
<destination>/service::A2/20210730T120000/log.log

However, if I want to rsync all files from serviceA from a particular day, I need a second wildcard like so: rsync -arv <remote-server>:/logs/service::A*/20210730* <destination>

However, this way I lose the top level directories and gives me the following structure at destination:

<destination>/20210730T120000/log.log
<destination>/20210730T120000/log.log

I would still like the service::<xx> level in my destination. It seemed like the way to achieve that was through the --include/--exclude flags.

However I tried the method from here but that didn't find the files:

rsync -av --include='service::A**/20210730**' --exclude='*' <remote-server>:/logs/ <destination>

receiving incremental file list
./
1
  • There's -R / --relative btw.
    – Tom Yan
    Aug 1, 2021 at 6:53

1 Answer 1

3

You will most likely want to use a combination of the inclusion patterns /service::A*/20210730*/** and */, and the exclusion pattern *. You may also want to include -m (--prune-empty-dirs) to only create the directory structure needed to hold the files that you are actually transferring.

rsync -avm \
    --include='/service::A*/20210730*/**' \
    --include='*/' \
    --exclude='*' \
    remote:/logs/ local-path/logs

In the file-listing phase of the transfer, this would traverse the whole hierarchy rooted at /logs on the remote host, including each directory (due to the */ inclusion pattern) and all files and directories matching the first pattern, but excluding anything else. That list of pathnames of directories and files would then be pruned of empty directories before the transfer actually started ("empty" means "containing no files matching any inclusion pattern").

The */ pattern is needed to allow rsync to enter the directories matching /service::A* etc. since this would otherwise not be possible due to the * exclusion pattern.


Since the above solution requires a complete traversal of everything under /logs, this may be slow if you have huge file hierarchies there.

If so, you may speed it up by being more selective with your inclusion patterns:

rsync -avm \
    --include='/service::A*/' \
    --include='/service::A*/20210730*/***' \
    --exclude='*' \
    remote:/logs/ local-path/logs

This explicitly allows rsync to enter the directories matching /service::A*/. It then includes all the timestamped directories thereunder that we are interested in, while excluding everything else.

A pattern like dir/*** differs from dir/** in that the former also includes the dir directory itself.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .