1

I have created the follow bash script to be used in cron for automatically backing up certain folders while also excluding certain sub folders:

folders_to_backup=(
    "/first/folder/to/backup/"
    "/second/folder/to/backup/"
)

backup_temp="/home/myuser/temp_backup_folder/"
mkdir $backup_temp
for i in "${folders_to_backup[@]}"
do
    cp -r $i $backup_temp
done

find $backup_temp | tar -czf "/home/myuser/backup.tgz" -T - -- exclude=*/venv --exclude=*/__pycache__
rm -r $backup_temp

So basically what I'm doing is creating a temporary folder, then copying all the relevant stuff I want to backup into that folder. Then I tar that folder (while excluding certain sub folders) and save it somewhere else. Then I delete the temporary folder.

The copying of everything into the temporary folder works correctly. However when I open the tar file it contains several duplicates of each file. For example if I backed up a file called test.txt then that file will be located in the temporary directory. However, when looking in the tar file, there will be multiple copies of test.txt

I can't seem to figure out what is going on between the temp folder through to the tar. I'm open to suggestions that correct the above issue or suggestions on how to do this without using a temp folder.

  • What backup method are you using? – Faheem Mitha Jun 4 '16 at 11:13
2

You need to either use tar --no-recursion or find ... ! -type d because each time your find is listing a directory, tar is recursively copying all of it.

Also, couldn't you simply avoid the copy and do a -C dir . for each dir in your array, eg:

args=()
for f in "${folders_to_backup[@]}"
do    args+=("-C" "$f" ".")
done
tar -czf "/home/myuser/backup.tgz" "${args[@]}" --exclude ...
0

I think all you need is rsync

For example:

$ rsync -auxnv -T /temp/dir --exclude={"/dir/1","/dir/2",...} \
  /source/file user@host:/dest/dir/

There is also an option for compression during transfer, -z

Please read man rsync

Check this too.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.