4

I have spent all day experimenting with rsync exclude-from and include-from, no success and completely lost. None of the examples or help I can find answer my exact, but quite simple requirement, and, either I can't make them work, or I do not get the result I want, nor can I understand why.

I am trying to set up an automatic daily backup, and after a few previous days attempts, based on this very good web site, which I have followed closely for incremental backups, and everything working well in tests, I have reached the stage of a dry-run of just my /home/Harry directory, without exclude-from and include-from, and it too works fine. I have also got the cron jobs sorted out that will run it for me.

Before anyone suggests using other than rsync and cron, note that I will be timing this to run daily at 11-30 p.m., with the computer shutting down a few minutes later. This too works fine in tests. If the computer is not on at that time it will not matter as there will be nothing to back up that day.

To progress further, and using the recommendations in "A Practical Guide to Fedora and Red Hat LInux", by Mark Sobell, 7th edition page 607, and other sources I want to refine it to include /home/, /var/, /usr/local/ and /etc/; and exclude /tmp/, /var/tmp/, /usr/tmp/, /proc/,/sys/ and /var/cache/.

Please: first of all are these sensible lists? and second: How do I specify this requirement in my rsync call?


Added after my question was answered.

Thanks to all who responded to my original question, I now have a working solution, listed in the script below. My experiences may be useful to anyone attempting something similar, and Michael Horner requested that I show the things that went wrong. Please refer to the website quoted for a complete description of how the cycle works, with a minimum of storage space needed.

My previous trials are, sort of, recorded in commented out lines in the script, which is what I used for my tests, originally to backup a few test files, and later to try to get the backup on to external hard drives. My practice is that, when I change something, I record the change by retaining and commenting out the original, so that I can go back to it again if necessary.

The long identifier starting CA6 is for a one Terabyte external hard drive on which I intend to store daily backups, probably with a 10 day cycle. I also have a 750 Gigabyte drive for weekly backups with probably a yearly cycle. You will see near the end of the script that this cycling has still to be attended to.

I am not concerned with any need to exactly restore the system if there is a crash. I have had three previous computers, none of them failed, but I kept manual backups on external hard drives and just use them, now, as a source for any material on them that I still need. So I would start with a bare installation and add what I need as I need it.

As hinted in my question, I experimented with using rsync exclude-from and include-from, but could not get the hang of the format of the necessary files. I might try to put the long list into a file, later.

#!/bin/bash
# Adapted from: http://webgnuru.com/linux/rsync_incremental.php
#  Website Backup Script
#======================================================================
# Define Variables
# Todays date in ISO-8601 format e.g. 2013-10-19:
DAY0=`date -I` 
# Yesterdays date in ISO-8601 format:
DAY1=`date -I -d "1 day ago"`
# The source directory:
SRC="/home/Harry/testrsync/bravo/"
#SRC="/home/Harry/"
# Filename of list of files to include
INC="/home/Harry/testrsync/include"
# Filename of file list to exclude
EXC="/home/Harry/testrsync/exclude"
# The target directory:
#TRG="/home/Harry/testrsync/backups/$DAY0"
TRG="/run/media/Harry/CA6C321E6C32062B/annals/$DAY0"
# The link destination directory:
#LNK="/home/Harry/testrsync/backups/$DAY1"
LNK="/run/media/Harry/CA6C321E6C32062B/$DAY1"
# The log file
#LOG="--logfile="/home/Harry/testrsync/backups/bulog --log-file-format="%t\n";""
LOG="--log-file=/home/Harry/testrsync/backups/bulog"
#The rsync options:
#OPT="--dry-run -avh --delete --link-dest=$LNK"
OPT="-aAXvh --delete --link-dest=$LNK"
#OPT="-ah --delete --link-dest=$LNK"
#======================================================================
#Execute the backup
#rsync -avvv $OPT --include-from=$INC --exclude-from=$EXC $SRC $TRG $LOG
#rsync -avv $OPT --include-from=$INC $SRC $TRG $LOG
#rsync $OPT--delete --stats --exclude-from=$EXC /home/Harry /var /usr/local /etc $TRG $LOG
rsync $OPT --exclude {"/dev/*","/proc/*","/sys/*","/tmp/*","/usr/tmp/*","/run/*","/mnt/*","/media/*","/var/cache/*","var/tmp/*","/","/lost+found"} /* $TRG $LOG
#Delete old backups cyclically
# for my tests I am going to use a three day cycle
DAY4=`date -I -d "4 days ago"`
#Delete the backup from 4 days ago, if it exists
if [ -d /home/Harry/testrsync/backups/$DAY4 ]
then
rm -r /home/Harry/testrsync/backups/$DAY4
#rm -r /run/media/HarryCA6C321E6C32062B/$DAY4
fi
# Cron does not output to a screen, so no result from next line
# zenity --info --text='Backup complete' --title="Backup Test"
  • 1
    Please post the commands you are using that aren't doing what you want. – Michael Homer Sep 30 '14 at 19:01
2

So, the --exclude-from file, will be:

var/tmp/
var/cache/

and use rsync like this:

rsync -aAv --delete --stats --exclude-from /excludes /home /var /usr/local /etc /Backup/
  • 1
    Shouldn't that be --exclude-from=FILE rather than --exclude-from FILE? Maybe it will accept both, but Linux man page mentions the equal sign. – peterph Sep 30 '14 at 19:25
  • 1
    Definitely accepts both invocations. – Scyld de Fraud Sep 30 '14 at 19:30
2

Include / (the root filesystem), and include the root of any other filesystem you want to back up (for example, if /home is on a separate partition, include it as well). Use the -x option to exclude all other filesystems: in-memory filesystems like /proc and /sys, mounted removable media, remote filesystems, etc.

Exclude the files you don't want to back up. /tmp is a common candidate, maybe also /var/tmp, some directories under /var/cache, etc.

Backup space is expensive, so you may not want to back up easily-restored files such as programs installed from packages. So you should exclude /usr except /usr/local. See Rsync filter: copying one pattern only for an introductory guide to rsync include and exclude lists. You'll want something like

rsync -ax --exclude='/tmp/*' --include=/usr/local --exclude=/usr / backup:
  • I thought / was on its own filesystem specifically because it doesn't need to be backed up? I would back up /etc, but isn't everything else very easily reinstallable? So that you just need /home, and maybe /etc to be thorough? (???) – Wildcard Oct 14 '15 at 6:16
  • @Wildcard You definitely want to back up /etc, and it has to be on the root filesystem since that's where the list of other filesystems to mount is stored. – Gilles Oct 14 '15 at 11:53
1

You can take full system backup using rsync. You can use --exlude=PATTERN for all directories that do not required .

rsync -- a fast, versatile, remote (and local) file-copying tool

  -a, --archive               archive mode; equals -rlptgoD (no -H,-A,-X)
  -A,  --acls                 preserve ACLs (implies -p)
  -X, --xattrs                preserve extended attributes
  -v, --verbose               increase verbosity

Example :

  rsync -aAXv --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/usr/tmp/*","/run/*","/mnt/*","/media/*","/var/cache/*","/","/lost+found"} /* /path/to/some/backup/folder

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.