2

How can I transfer data by excluding files over 100MB but including files that are over 100MB if they match a pattern of known file extensions?

I've read through rsync options and I don't think I can achieve this with rsync, because --max-size= is not flexible like this, even in combination with --include or --exclude.

3

In two steps (for simplicity, even though these steps can definitely be combined).

First transfer "small" files:

find /source/path -type f -size -100M -print0 |
rsync -av -0 --files-from=- / user@server:/destination/

Then transfer "big" files whose filenames match pattern:

find /source/path -type f -size +99M -name 'pattern' -print0 |
rsync -av -0 --files-from=- / user@server:/destination/

This is, however, untested.

-print0 in GNU find (and others) will print the found names with a nul delimiter, and -0 with rsync will make --files-from- interpret this standard input stream in that particular way.

The file paths read with --files-from should be relative to the specified source, that's why I use / as the source in rsync (I'm assuming /source/path in find is an absolute path).


Combined variation (also not tested):

find /source/path -type f \
    \( -size -100M -o -name 'pattern' \) -print0 |
rsync -av -0 --files-from=- / user@server:/destination/

With more than one allowable pattern string for "big" files:

find /source/path -type f \
    \( -size -100M -o -name 'pattern1' -o -name 'pattern2' -o -name 'pattern3' \) -print0 |
rsync -av -0 --files-from=- / user@server:/destination/

Each pattern may be something like *.mp4 or whatever file extensions you use. Note that these needs to be quoted, as in -name '*.mp4'.

  • I'm not too sure why you have used \( and \) inbetween the -size and final pattern, but it works with and without? Anyway, this example is still picking up files that do not match the pattern. – adampski Feb 15 '18 at 8:37
  • @adampski Are you referring to my last example? That would trigger for files of size less than 100 Mb, or for file whose names matches any of the patterns. The \( ... \) groups the conditional so that -print0 happens for any file that matches any of the conditions. – Kusalananda Feb 15 '18 at 8:40
  • 1
    Ah, I understand now. I noticed it was still -100M so I assumed this was a mistake and made it +99M. However, the "or" in your response is the key bit of information that enables me to understand that it's copying all files less than 100M or files that match the pattern. Thank you! – adampski Feb 15 '18 at 8:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.