Copy all files with the directory structure since the last run on remote server

Question

I would like to scan for my /tmp/ directory now and then and check what files have been changed since I last time did it. All files that have been changed should be copied to remote server (as a backup), perserving the directory structure. I tried something like this, but it's not quite working on Ubuntu, throwing Permission denied (publickey).

#!/bin/bash
TARGET=/bkp
[email protected]
SOURCE=/tmp

touch $TARGET/last_sync

rsync \
    -ahrv \
    --update \
    -e \
    --files-from=<ssh -i /home/ubuntu/ssh_key.pem $HOST "find $SOURCE -type f -newer $SOURCE/last_sync -exec realpath --relative-to=$SOURCE '{}' \;" \
    $HOST:$SOURCE \
    $TARGET

rsync -ahv $TARGET/last_sync $HOST:$SOURCE

What I had in plan is to have on a source machine:

ubuntu@source:~/$ tree /tmp
.
├── file1.txt
├── dir1
│   ├── subdir1
│   │   └── file3.txt
│   └── file2.txt
├── file4.txt

and on the first run it should be copied on a remote machine as

ubuntu@remote:~/$ tree /bkp
.
├── file1.txt
├── dir1
│   ├── subdir1
│   │   └── file3.txt
│   └── file2.txt
├── file4.txt

Several hours later, new file /tmp/dir1/file5.txt was added, and file /tmp/file4.txt was changed

After the script was executed, it should be synced (added both files) on remote:

ubuntu@remote:~/$ tree /bkp
.
├── file1.txt
├── dir1
│   ├── subdir1
│   │   └── file3.txt
│   └── file2.txt
|   | -- file5.txt <-- ADDED
├── file4.txt <-- ADDED (replaced)

I was able to achieve something similar on macos -> ubuntu, but with scp and more "logic" behind it, was checking files changed in the last N hours and running the script every N hours..

Answer 1

The primary issue seems to be that you're getting a Permission denied (publickey) error. This is typical of an authentication issue when using certificate-based authentication through ssh. Looking at your code you invoke ssh three times but only once do you provide a non-default private certificate (key):

1. rsync to perform an update
2. ssh to invoke find to look for newer files
3. rsync to copy a synchronisation beacon file

If /home/ubuntu/ssh_key.pem is the appropriate key to use for this connection, you need to reference it each time.

A secondary issue is that the syntax looks really wrong for much of your code. I know you took my recommendation on ServerFault to put the code through https://shellcheck.net/, but there are still places where the code can now be parsed by bash but is still syntactically wrong. For example,

rsync \
    -ahrv \
    --update \
    -e \
    --files-from=<ssh -i /home/ubuntu/ssh_key.pem $HOST "find $SOURCE -type f -newer $SOURCE/last_sync -exec realpath --relative-to=$SOURCE '{}' \;" \
    $HOST:$SOURCE \
    $TARGET

If we rewrite this so that the options taking arguments are linked together, we get this:

rsync \
    -ahrv \
    --update \
    -e --files-from=
    -i \
    /home/ubuntu/ssh_key.pem $HOST "find $SOURCE -type f -newer $SOURCE/last_sync -exec realpath --relative-to=$SOURCE '{}' \;" $HOST:$SOURCE $TARGET
    <ssh

• There is no file ssh from which to read stdin, nor is there anything to process it
• The command --files-from= referenced by -e does not exist
• The four source files/directories would be copied to $TARGET (if the rest of the command had been valid)

I believe I can see what you're trying to achieve, but I'm not entirely sure why you don't just use rsync in its normal mode of operation and have it copy files directly. No need for find. No need for a synchronisation beacon file.

#!/bin/bash
[email protected]
source=/path/to/source
target=/bkp

rsync -av -e 'ssh -i /home/ubuntu/ssh_key.pem' "$rhost:$source/" "$target"