What should I include and exclude with rsync to back up my system?

Question

I have spent all day experimenting with rsync exclude-from and include-from, no success and completely lost. None of the examples or help I can find answer my exact, but quite simple requirement, and, either I can't make them work, or I do not get the result I want, nor can I understand why.

I am trying to set up an automatic daily backup, and after a few previous days attempts, based on this very good web site, which I have followed closely for incremental backups, and everything working well in tests, I have reached the stage of a dry-run of just my /home/Harry directory, without exclude-from and include-from, and it too works fine. I have also got the cron jobs sorted out that will run it for me.

Before anyone suggests using other than rsync and cron, note that I will be timing this to run daily at 11-30 p.m., with the computer shutting down a few minutes later. This too works fine in tests. If the computer is not on at that time it will not matter as there will be nothing to back up that day.

To progress further, and using the recommendations in "A Practical Guide to Fedora and Red Hat LInux", by Mark Sobell, 7th edition page 607, and other sources I want to refine it to include /home/, /var/, /usr/local/ and /etc/; and exclude /tmp/, /var/tmp/, /usr/tmp/, /proc/,/sys/ and /var/cache/.

Please: first of all are these sensible lists? and second: How do I specify this requirement in my rsync call?

---

Added after my question was answered.

Thanks to all who responded to my original question, I now have a working solution, listed in the script below. My experiences may be useful to anyone attempting something similar, and Michael Horner requested that I show the things that went wrong. Please refer to the website quoted for a complete description of how the cycle works, with a minimum of storage space needed.

My previous trials are, sort of, recorded in commented out lines in the script, which is what I used for my tests, originally to backup a few test files, and later to try to get the backup on to external hard drives. My practice is that, when I change something, I record the change by retaining and commenting out the original, so that I can go back to it again if necessary.

The long identifier starting CA6 is for a one Terabyte external hard drive on which I intend to store daily backups, probably with a 10 day cycle. I also have a 750 Gigabyte drive for weekly backups with probably a yearly cycle. You will see near the end of the script that this cycling has still to be attended to.

I am not concerned with any need to exactly restore the system if there is a crash. I have had three previous computers, none of them failed, but I kept manual backups on external hard drives and just use them, now, as a source for any material on them that I still need. So I would start with a bare installation and add what I need as I need it.

As hinted in my question, I experimented with using rsync exclude-from and include-from, but could not get the hang of the format of the necessary files. I might try to put the long list into a file, later.

#!/bin/bash
# Adapted from: http://webgnuru.com/linux/rsync_incremental.php
#  Website Backup Script
#======================================================================
# Define Variables
# Todays date in ISO-8601 format e.g. 2013-10-19:
DAY0=`date -I` 
# Yesterdays date in ISO-8601 format:
DAY1=`date -I -d "1 day ago"`
# The source directory:
SRC="/home/Harry/testrsync/bravo/"
#SRC="/home/Harry/"
# Filename of list of files to include
INC="/home/Harry/testrsync/include"
# Filename of file list to exclude
EXC="/home/Harry/testrsync/exclude"
# The target directory:
#TRG="/home/Harry/testrsync/backups/$DAY0"
TRG="/run/media/Harry/CA6C321E6C32062B/annals/$DAY0"
# The link destination directory:
#LNK="/home/Harry/testrsync/backups/$DAY1"
LNK="/run/media/Harry/CA6C321E6C32062B/$DAY1"
# The log file
#LOG="--logfile="/home/Harry/testrsync/backups/bulog --log-file-format="%t\n";""
LOG="--log-file=/home/Harry/testrsync/backups/bulog"
#The rsync options:
#OPT="--dry-run -avh --delete --link-dest=$LNK"
OPT="-aAXvh --delete --link-dest=$LNK"
#OPT="-ah --delete --link-dest=$LNK"
#======================================================================
#Execute the backup
#rsync -avvv $OPT --include-from=$INC --exclude-from=$EXC $SRC $TRG $LOG
#rsync -avv $OPT --include-from=$INC $SRC $TRG $LOG
#rsync $OPT--delete --stats --exclude-from=$EXC /home/Harry /var /usr/local /etc $TRG $LOG
rsync $OPT --exclude {"/dev/*","/proc/*","/sys/*","/tmp/*","/usr/tmp/*","/run/*","/mnt/*","/media/*","/var/cache/*","var/tmp/*","/","/lost+found"} /* $TRG $LOG
#Delete old backups cyclically
# for my tests I am going to use a three day cycle
DAY4=`date -I -d "4 days ago"`
#Delete the backup from 4 days ago, if it exists
if [ -d /home/Harry/testrsync/backups/$DAY4 ]
then
rm -r /home/Harry/testrsync/backups/$DAY4
#rm -r /run/media/HarryCA6C321E6C32062B/$DAY4
fi
# Cron does not output to a screen, so no result from next line
# zenity --info --text='Backup complete' --title="Backup Test"

Answer 1

Include / (the root filesystem), and include the root of any other filesystem you want to back up (for example, if /home is on a separate partition, include it as well). Use the -x option to exclude all other filesystems: in-memory filesystems like /proc and /sys, mounted removable media, remote filesystems, etc.

Exclude the files you don't want to back up. /tmp is a common candidate, maybe also /var/tmp, some directories under /var/cache, etc.

Backup space is expensive, so you may not want to back up easily-restored files such as programs installed from packages. So you should exclude /usr except /usr/local. See Rsync filter: copying one pattern only for an introductory guide to rsync include and exclude lists. You'll want something like

rsync -ax --exclude='/tmp/*' --include=/usr/local --exclude=/usr / backup:

Answer 2

You can take full system backup using rsync. You can use --exlude=PATTERN for all directories that do not required .

rsync -- a fast, versatile, remote (and local) file-copying tool

  -a, --archive               archive mode; equals -rlptgoD (no -H,-A,-X)
  -A,  --acls                 preserve ACLs (implies -p)
  -X, --xattrs                preserve extended attributes
  -v, --verbose               increase verbosity

Example :

  rsync -aAXv --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/usr/tmp/*","/run/*","/mnt/*","/media/*","/var/cache/*","/","/lost+found"} /* /path/to/some/backup/folder

Answer 3

So, the --exclude-from file, will be:

var/tmp/
var/cache/

and use rsync like this:

rsync -aAv --delete --stats --exclude-from /excludes /home /var /usr/local /etc /Backup/

Answer 4

If you want to copy all your system into another server, so the other server will be a running copy of your system, use this command:

rsync -aAX --del --info=progress2 /mnt/source/ /mnt/target/ --exclude={/dev/*,/proc/*,/sys/*,/tmp/*,/run/*,/mnt/*,/media/*,/lost+found,/boot/*,/var/tmp/*,/var/cache/*,/usr/tmp/*}

-a, --archive This is equivalent to -rlptgoD. It is a quick way of saying you want recursion and want to preserve almost everything. Note that -a does not preserve hardlinks, because finding multiply-linked files is expensive. You must separately specify -H.

-A, --acls causes rsync to update the destination ACLs to be the same as the source ACLs.

-X, --xattrs This option causes rsync to update the destination extended attributes to be the same as the source ones.

--delete-during, --del Request that the file-deletions on the receiving side be done incrementally as the transfer happens.

Answer 5

Based off of Which Folders To Include In backup?, I chose to exclude the folders/files shown:

#!/bin/bash

sudo rsync -aAXvlH --delete --exclude={"/bin/*","/boot/*","/dev/*","/home/*","/lib/*","/lib32/*","/lib64/*","/media/*","/mnt/*","/NAS/*","/Pool-1tb/*","/Pool-500gb/*","/proc/*","/run/*","/sbin/*","/srv/*","/sys/*","/tmp/*","/usr/bin/*","/usr/games/*","/usr/include/*","/usr/lib/*","/usr/lib32/*","/usr/libexec/*","/usr/sbin/*","/usr/share/*","/usr/src/*",".Trash*"} / /NAS/System\ Files/

Here are the flags I used:

-a - Archive mode includes among other things, the options -o and -g, which preserves owners and groups. This requires that you run rsync as root.
-A - Preserve Access Control List.
-X - Preserve extended attributes.
-v - It will show the progress of the backup.
-l - Copy symlinks as symlinks.
-H - Preserve hard links.

Then I created a cron job for a file containing the above code.

To resolve the plethora of symlink errors I got when running rsync, on my Synology, I had to go to Control Panel / File Services / SMB / Advanced Settings and check the box next to "Allow symbolic links within shared folders". You may have a similar setting depending on which NAS you are using.

If you found this helpful, please consider an up-vote. Any constructive criticism is also appreciated. Thanks!

Answer 6

In case anyone else finds this, consider using

rsnapshot

which in turn uses rsync. It is robust and proven.