13

I want to make a backup of a directory that contains thousands of sub-directories and deep sub-folder paths. In many of these directories there are a big number of files starting with .. I want to make sure that all of the . files in sub-directories and sub-sub-directories and etc. get properly copied. Is it enough to specify include="*"? Will this cover everything?

rsync -rvh --compress-level=0 --stats --progress --include ".*"  user@vm:/mnt/storage8/backups ~/data/backup_of_backups/

3 Answers 3

9

All files are included by default, so if you want to include all files, don't pass any --include or --exclude option.

If you do use patterns, rsync doesn't treat dot files specially. If you wanted to skip dot files, you'd have to explicitly use --exclude='.*'.

Note that --include='.*' would only include dot files. This is a shell pattern, where . stands for itself and * means “any sequence of characters”, not a regex where . means “any character” and * means “any number of the preceding character or group”. Without any exclude directive, you still get all files, so an include directive is just pointless, but if you had some exclude directives, --include='.*' would not mean “include all files including dot files”, it would only mean “include dot files” (and on its own it wouldn't recurse into directories whose name don't start with a dot).

0
5

Without specifying any inclusion or exclusion patterns, rsync will copy hidden files and directories.

rsync --stats --progress -av user@vm:/mnt/storage8/backups ~/data/backup_of_backups/
-1

Use shopt -s dotglob. It includes the dot files. It's a bash setting so will fix mv command too.

https://unix.stackexchange.com/a/446353/346155

EDIT: the accepted answer correctly says rsync includes dot files by default. As this question is the top result from Google for "rsync dot files", my rsync command had failed. Whereas the OP had not tried using plain rsync, I often put an asterisk after the slash which is added to the end of a directory to ensure the dir doesnt get copied into the target dir, which happens when assuming the 'sync' in rsync will actually sync the dirs. So in my case, using the shell option for dotglob fixed my problem, and might fix yours. It also works on other wildcard matching of the /dir/* kind. Personally, I think this is a more intuitive and explicit than using just the /, as just /dir/ could imply just completing a long list of dir separators, as it is interpreted when used with the target dir.

Further, I wonder if using this shell option by default would have any detrimental side effects, as it is not imposed in Bash by default. It's just part of the Unix design, I assume, that hidden files are not supposed to be copied or moved by default (again, unless using the /dir/ form). I would advocate for making all files copied the default, as that seems more intuitive and expected, unless there is a detrimental reason I'm missing (the only thing I can think of is having location specific information that one wouldnt want moved, or just wanting to copy media data from a dir instead of any config files - which I might argue are better to have copied than not).

6
  • Quoted shell patterns are not expanded by the shell, the mv command is not being used here. It is unclear how you would apply your solution to the user's issue (the linked answer seems irrelevant, other than that it uses rsync).
    – Kusalananda
    Mar 29, 2022 at 6:11
  • they, you must not understand "I want to make sure that all of the . files in sub-directories and sub-sub-directories and etc. get properly copied". That is exactly what shopt -s dotglob does. That it also works on mv is a bonus. (thanks for downvoting a correct solution without specifying exactly why you think it is incorrect. Very awesome of you. Posting incomplete criticism must be how you get 12k rep in 6 months).
    – alchemy
    Mar 29, 2022 at 17:51
  • Hmm... I thought this would be easy to fix. Just show what the user should do. Saying "Use shopt -s dotglob" clearly does not solve the issue for the user (it wouldn't change the way their command would work). Saying "it will fix the mv command" is nonsensical as the user is not using the mv command. I'm just trying to get you to make your answer relevant to the question. If you believe the question is a duplicate, then mark it as such instead of posting a link.
    – Kusalananda
    Mar 29, 2022 at 17:59
  • Listen, making the bash shell glob hidden names does not help the user. The command they use does not do globbing (because, as I mentioned in my first comment, the shell won't do globbing on quoted patterns). We can't be sure they even use the bash shell. If you want to use the dotglob shell option in bash in your answer, show how. Also note that the result of a glob operation can't be used with an external command if too many names match the pattern.
    – Kusalananda
    Mar 29, 2022 at 18:14
  • Okay, in fairness, you did try to say the OP wasnt using a wildcard that wasnt quoted. I was trying to quickly add my solution to the first Google hit I came to so I could get back to work. I assumed the OP had at least tried using his command, which if I'm not mistaken would have worked anyway. I do think others will land here like myself, so I clarified my post. The answer is that there is no problem actually. But for others trying to understand why dot files are not copied when using the /dir/* for, this post should be helpful. I also make a larger point about this linux/bash default.
    – alchemy
    Mar 29, 2022 at 19:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .