I want to make a backup of a directory that contains thousands of sub-directories and deep sub-folder paths. In many of these directories there are a big number of files starting with .. I want to make sure that all of the . files in sub-directories and sub-sub-directories and etc. get properly copied. Is it enough to specify include="*"? Will this cover everything?
rsync -rvh --compress-level=0 --stats --progress --include ".*" user@vm:/mnt/storage8/backups ~/data/backup_of_backups/
All files are included by default, so if you want to include all files, don't pass any --include or --exclude option.
If you do use patterns, rsync doesn't treat dot files specially. If you wanted to skip dot files, you'd have to explicitly use --exclude='.*'.
Note that --include='.*' would only include dot files. This is a shell pattern, where . stands for itself and * means “any sequence of characters”, not a regex where . means “any character” and * means “any number of the preceding character or group”. Without any exclude directive, you still get all files, so an include directive is just pointless, but if you had some exclude directives, --include='.*' would not mean “include all files including dot files”, it would only mean “include dot files” (and on its own it wouldn't recurse into directories whose name don't start with a dot).
Without specifying any inclusion or exclusion patterns, rsync will copy hidden files and directories.
rsync --stats --progress -av user@vm:/mnt/storage8/backups ~/data/backup_of_backups/
Use shopt -s dotglob. It includes the dot files. It's a bash setting so will fix mv command too.
https://unix.stackexchange.com/a/446353/346155
EDIT: the accepted answer correctly says rsync includes dot files by default. As this question is the top result from Google for "rsync dot files", my rsync command had failed. Whereas the OP had not tried using plain rsync, I often put an asterisk after the slash which is added to the end of a directory to ensure the dir doesnt get copied into the target dir, which happens when assuming the 'sync' in rsync will actually sync the dirs. So in my case, using the shell option for dotglob fixed my problem, and might fix yours. It also works on other wildcard matching of the /dir/* kind. Personally, I think this is a more intuitive and explicit than using just the /, as just /dir/ could imply just completing a long list of dir separators, as it is interpreted when used with the target dir.
Further, I wonder if using this shell option by default would have any detrimental side effects, as it is not imposed in Bash by default. It's just part of the Unix design, I assume, that hidden files are not supposed to be copied or moved by default (again, unless using the /dir/ form). I would advocate for making all files copied the default, as that seems more intuitive and expected, unless there is a detrimental reason I'm missing (the only thing I can think of is having location specific information that one wouldnt want moved, or just wanting to copy media data from a dir instead of any config files - which I might argue are better to have copied than not).
mv command is not being used here. It is unclear how you would apply your solution to the user's issue (the linked answer seems irrelevant, other than that it uses rsync).
shopt -s dotglob does. That it also works on mv is a bonus. (thanks for downvoting a correct solution without specifying exactly why you think it is incorrect. Very awesome of you. Posting incomplete criticism must be how you get 12k rep in 6 months).
shopt -s dotglob" clearly does not solve the issue for the user (it wouldn't change the way their command would work). Saying "it will fix the mv command" is nonsensical as the user is not using the mv command. I'm just trying to get you to make your answer relevant to the question. If you believe the question is a duplicate, then mark it as such instead of posting a link.
bash shell glob hidden names does not help the user. The command they use does not do globbing (because, as I mentioned in my first comment, the shell won't do globbing on quoted patterns). We can't be sure they even use the bash shell. If you want to use the dotglob shell option in bash in your answer, show how. Also note that the result of a glob operation can't be used with an external command if too many names match the pattern.
/dir/* for, this post should be helpful. I also make a larger point about this linux/bash default.