0
  • I have two remote data acquisition machines that are continuously writing data to their own local data.txt files.
  • The remote machines begin acquiring data and appending their data.txt with new data upon powering up/reboot via a systemd service.
  • The host machine enables/starts the service on each remote machine.

I would like to periodically and automatically transfer their respective data files to my host machine using a shell script.

Note: I currently only have access to one remote machine, and will have access to the second in the future. I'm trying to prepare for when this machine comes online. This is why I can't test out my "thoughts" on how to accomplish this.

Here is what I have cooked up to accomplish this for a single remote machine:

#!/usr/bin/env bash

#Enable data acquisition service on remote machine
REMOTE1="user@remote1"
REMOTE2="user@remote2"

ssh $REMOTE1 "systemctl start my.service"
while [ $? -ne 0 ]; do !!; done # waits for service to start, returns 0

ssh $REMOTE2 "systemctl start my.service"
while [ $? -ne 0 ]; do !!; done

# transfer data file every 60 seconds, until SIGINT from host
while sleep 60; do scp $REMOTE1:/data1.txt /host/; done

exit 0

I have run this successfully using one remote machine; I have not noticed any missing data/corruption either. To get this working with two remote machines, my first thought is to add another instance of the while loop:

while sleep 60; do scp $REMOTE1:/data1.txt /host/; done
while sleep 60; do scp $REMOTE2:/data2.txt /host/; done

However, I can see this may not work... given that the first loop will never break to begin the second one (?). My second thought would be to use a single while loop, but adding the another scp command for the second machine:

while sleep 60; do scp $REMOTE1:/data1.txt /host/; scp $REMOTE2:/data2.txt /host/; done

I am aware that this method could end up "drifting" if the command execution time varies. Though I am not worried about the transfer occurring exactly every X amount of time, it just needs to happen (sudo) periodically. Would love to hear comments/suggestions on how to accomplish this.

4
  • while [ $? -ne 0 ]; do !!; done -- are you relying on history expansion here? Does it work? Because Bash's manual says it's disabled in interactive shells by default, but I can't seem get it manually enabled either...
    – ilkkachu
    May 12 at 21:44
  • 1
    Actually, I'm not sure what you're asking at all. You mention that sleeping a constant amount of time might end up having the exact time drift, but you seem to be okay with that. I'm not sure what you mean with the "(sudo)" in parenthesis, though.
    – ilkkachu
    May 12 at 21:49
  • @ilkkachu while [ $? -ne 0 ]; do !!; done does work for me. Using Ubuntu 16.04
    – earl
    May 13 at 13:16
  • ok, I tried that on Debian and a few different versions of Bash, and it only works if I explicitly enable set -o history and set -o histexpand/set -H. Both are off by default, but I have no idea if that's changeable at compile-time, or if Ubuntu 16.04 in particular does something different there. In any case, I'm not exactly sure what you're asking about here.
    – ilkkachu
    May 13 at 14:34

1 Answer 1

1

Use an array to contain the pair of hosts. Initially the array can contain just a single element, and you can add the second host later. Here I've included two sets of credentials so you can see the syntax. For the copying I've chosen to use rsync but you could revert to scp if you preferred.

#!/bin/bash

# Remote credentials
remotes=('user@remote1' 'user@remote2')

# Enable data acquisition service
for remote in "${remotes[@]}"
do
    ssh -n "$remote" 'systemctl start my.service'
done

# Transfer data file with an interval of 60 seconds, until SIGINT
while sleep 60
do
    for remote in "${remotes[@]}"
    do
        rsync -tz --contimeout=10 "$remote:/data1.txt" /host/
        # scp -pC "$remote:/data1.txt" /host/
    done
done

# Forever

The loop delays 60 seconds between executions, so if the copies take 15 seconds each you'll actually loop every 90 seconds. If you want the loop to aim for a minute cycle regardless of the copying duration you can use bash's built-in clock:

interval=60 start=$SECONDS
while sleep $(( interval - ((SECONDS - start) % interval) ))
do
    …
done

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.