2

I'm sure this has been been answered in bits and pieces on many sites, but somehow I can't seem to get this (as a whole) to work reliably.

Scenario: I have a server that I would like to do incremental backups on (I'm thinking daily).

The files and directories I want to backup are located in the directories: /home/ and /srv/samba/.

The backup should be .tar.gz to /mnt/backup/ directory, and the filename should be something like inc_backup_yyyymmdd.tar.gz.

After the .tar.gz is successfully created, the file should be sent to an off-site server using ssh.

Right now the server creates a FULL backup once a week, and I manually send it off-site at random intervals (the .tar.gz size is about 60 GB, so it takes a while...).

Ideally I would like to .tar.gz ONLY regular files and folders. Files starting with . and ~ should be excluded, as well as directories starting with .

There is one exception:
In /home/[user]/ and /srv/samba/[someshare]/ folders I have ".recycle" folders, that should be included in the backup, but the file exclusions should be applied.

E.g.

/home/user/.hiddendirectory/ (excluded)
/home/user/.hiddendirectory/myfile.txt (excluded)
/home/user/myfile.txt (included)
/home/user/.myfile.txt (excluded)
/home/user/~myfile.txt (excluded)
/home/user/mydirectory/myfile.txt (included)
/home/user/.recycle/ (included)
/home/user/.recycle/~tmpfile (excluded)
/home/user/.recycle/.myfile (excluded)
/home/user/.recycle/myfile.txt (included)

I get started by using find:

Please note that the following commands include non-standard operators (-not, -or), so if used, please ensure your system supports them

find /home -not -path '*/\.*' ! \( -name '.*' -or -name '~*' \) -mtime -1

This seems to find the files and folders I need. But immediately that I pipeline this to tar, it seems to ignore all the exclusions?

find /home -not -path '*/\.*' ! \( -name '.*' -or -name '~*' \) -mtime -1 -print0 | tar -zcvf /tmp/testbackup1.tar.gz --null -T -

I seem to get the correct result using the following:

find /home -not -path '*/\.*' ! \( -name '.*' -or -name '~*' \) -mtime -1 -print0 | tar -Avf /tmp/testbackup1.tar | gzip -c /tmp/testbackup1.tar > testbackup1.tar.gz

...but this seems overly complicated, and results in an empty tarball, although the tar has significant size? And none of these take into account the .recycle inclusion.

A bonus would be an encrypted tar.gz for added security.

Sending the file using SSH is the easy part.

My thoughts are to put this into a script file, that is executed by Cron.

The script would:

  • Create the backup file (handling the naming procedure)
  • Send the backup file over SSH to an off-site server
  • Delete old files (e.g. 2 weeks) from the .recycle directories

The (in place) full backup would be executed once a month, and will also handle cleanup of old backup files (2 months).

So, in the end I would have 2 full backups and approx 60 incremental backups. Once a full backup is dropped, the incremental backups (refereing to that) would also be deleted. And all the backup files will be located on-site and off site.

Any advice (even totally new approaches) is really appreciated!

Thanks in advance!

  • 1
    Why don't you use some of the well-established backup programs (attic, bup, obnam)? – Jan Oct 14 '15 at 20:01
  • 4
    why not to use rsync? – Jakuje Oct 14 '15 at 20:08
  • find has no operators called -not or -or. But if you like to use find and tar, why don't you just use star? – schily Oct 14 '15 at 20:15
  • Thank you for your ideas! All of those look like possible approaches, and I will have to do some research as to which approach suits my needs. @schily, That's weird. At least according to link there are -not and -or operators for expressions, and they seem to work in my example. – Sisu Oct 14 '15 at 20:36
  • 3
    You're reinventing the wheel, and your apparent aversion to using rsync is going to make that all the more painful. There are plenty of pre-existing solutions like Duplicity, but if you're hell-bent on rolling your own I strongly suggest using rsync instead of cobbling together find and tar. – Sammitch Oct 15 '15 at 1:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.