6

I have the following script using rsync for backuping and synchronization of my files.

How can I make this script more compact, faster and with better error handling?

#!/bin/sh
  
errors=""

sudo rsync -avh --delete --no-o --no-g /home/xralf/audio /media/extdevice/rsync_backups/
if [ "$?" -eq "0" ]
then
  echo "no errors in audio"
else
  errors="${errors}error in audio\n"
fi

sudo rsync -avh --delete --no-o --no-g /home/xralf/books /media/extdevice/rsync_backups/
if [ "$?" -eq "0" ]
then
  echo "no errors in books"
else
  errors="${errors}error in books\n"
fi

sudo rsync -avh --delete --no-o --no-g /home/xralf/source_code /media/extdevice/rsync_backups/
if [ "$?" -eq "0" ]
then
  echo "no errors in source_code"
else
  errors="${errors}error in source_code\n"
fi

# more such directories with this code pattern

echo ${errors}

Later I plan not to watch the script execution, so I'd like to see only the result and believe everything went right. I ran the script, but in the end, the only information I have is error in source_code, and I can't see, what exactly the error is (which file caused it).

4
  • Does it need to be sh or can you use bash?
    – terdon
    Commented Nov 1, 2022 at 21:03
  • @terdon I think I can use bash. I haven't scripted much in my life, so far, so don't know the pros and cons.
    – xralf
    Commented Nov 1, 2022 at 21:13
  • Possible duplicate unix.stackexchange.com/q/723309/100397 Commented Nov 2, 2022 at 15:44
  • @roaima This question was first. You solved for me the speed. Thanks a lot. Here I would like to solve a bit better error handling yet. The existing answer is on a good way.
    – xralf
    Commented Nov 2, 2022 at 18:43

4 Answers 4

5
+100

If your priority is to be compact and fast (ie: minimizing the number of processes, but in this case you are limited by your disk IOs, not by cpu or memory):

#!/bin/bash
sudo rsync -a --delete --no-o --no-g \
       /home/xralf/{audio,books,source_code} \
       /media/extdevice/rsync_backups/ &&
    echo "rsync completed successfully" 1>&2 || echo "rsync ended with errors" 1>&2

Removing the -v option will make the output less noisy, so that you can focus on explicit errors, if any. The 1>&2 redirects STDOUT to STDERR, so that error messages are sent where expected. If running unattended, the following alternatives may be better options than the built-in echo, at the cost of on extra process:

  1. a server can use logger to send the message to syslog for centralized handling. Exact syntax will depend of the server configuration.

  2. using date to add timing information, with the last line of the script becoming:

date +"[%Y/%m/%d %H:%M:%S] rsync completed successfully" 1>&2 || \
date +"[%Y/%m/%d %H:%M:%S] rsync ended with errors" 1>&2

to output something like [2022/12/05 14:37:27] rsync completed successfully

2
  • So, I have to check for errors something like this? tail /var/log/syslog?
    – xralf
    Commented Nov 5, 2022 at 21:01
  • 1
    @xralf If you use logger, the messages will be send to the syslog daemon. Depending of the priority and category you used when sending the message, it will be added in a specific file (or other specific actions, like sending message on a remote server). /var/log/syslog usually contains most messages. Check the manual for syslog.conf linux.die.net/man/5/syslog.conf for more info. While using syslog is a good practice for system management, it is a topic on itself, beyond the scope of the initial question.
    – Uriel
    Commented Nov 6, 2022 at 10:11
5

If you are not forced to use sh but can use a more complex shell like bash, you can do something like this:

#!/bin/bash

dirs=( "audio" "books" "source_code" );
errors=()
for dir in "${dirs[@]}"; do
  sudo rsync \
       -avh --delete --no-o --no-g \
       /home/xralf/"$dir" \
       /media/extdevice/rsync_backups/ &&
    echo "no errors in $dir" ||
      errors+=("error in $dir\n")
done

printf '%s\n' "${errors[@]}"

You probably also want to capture stderr but this does the same thing as your script.

6
  • Thanks. And is this model of error handling sufficient or is there a better way? Is it possible to make if faster? I heard about the --whole-file switch but don't understand what it does, so I'm afraid using it.
    – xralf
    Commented Nov 1, 2022 at 21:19
  • Later I plan not to watch the script execution, so I'd like to see only the result and believe everything went right. I would like to even enter the password only once (at the beginning). Now I have to repeatedly enter the password.
    – xralf
    Commented Nov 1, 2022 at 21:22
  • I ran the script, but in the end, the only information I have is error in source_code, and I can't see, what exactly the error is.
    – xralf
    Commented Nov 1, 2022 at 21:30
  • 1
    @xralf whether you can make it faster will depend very much on the kind of disks you are writing to/reading from, but sure, if your disk supports it you can parallelize the execution. To only enter the password once, remove sudo from the rsync commands and just run the script itself with sudo script.sh. As for the last, yes, that's why I said you probably want to capture the stderr but your original didn't do that and you didn't explain you needed it so I didn't know. Please edit your question and add all requirements there.
    – terdon
    Commented Nov 1, 2022 at 21:40
  • Thanks. I edited the question.
    – xralf
    Commented Nov 1, 2022 at 21:51
1

The rsync command accepts the parameter --log-file=FILE. You can examine it for errors when the backup is complete.

For efficiency's sake, it also accepts --exclude-from=FILE and --include-from=FILE. You can create text files containing a list of everything you do and do not want to back up and call them from a single rsync command rather than multiple ones.

See man rsync or the online man page.

0

Make a file called module.sh and give it the following text:

#!/usr/bin/env bash                                                                                                                     

# -A=Archive. Use recursion and preserve almost everything.
# -V=Verbose.
# -H=Preserve hard links.
# -Z=Compress.
# --Delete=
#     If node in directory 'from' gets deleted,
#     delete it in directory 'to'.
# --No-o=No Owner. 
# --No-g=No Group.
# The --No's are partially undoing the Archive option.

sync_dir(){
    local from="${1}";
    local to="${2}";

    rsync               \
        -avhz           \
        --delete        \
        --no-o          \
        --no-g          \
            "${from}/"  \
            "${to}"     \
        1>/dev/null     \
        ;

    # Store the sync success.
    local sync_result="${?}";

    # The project name is the last directory of the from string.
    local project_name=$(grep -oE '[^/]*$' <<<"${from}");

    local error="No error in: ${project_name}";
    if [ "${sync_result}" -ne "0" ]; then
        error="Error in: ${project_name}\n";
        fi  
    printf "${error}\n";
    };

Then you make another file and source the module from there:

#!/usr/bin/env bash
                                                                       
. ./module.sh;

sync_dir '/home/xralf/audio' '/media/extdevice/rsync_backups/';
# More sync dirs here.

Note different Shebang, this one is POSIX compliant.

I also added Rsync's -z option. It compresses the data en route.

2
  • You cannot call a function that is defined afterwards.
    – Uriel
    Commented Nov 6, 2022 at 11:09
  • Fixed. .........
    – john-jones
    Commented Nov 6, 2022 at 11:18

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .