2

I am currently writing a script that will perform backups on a set of databases, comress them, encrypt them, transfer them to a separate storage machine, and verify that the file made it successfully. I can create the backup and perform the standard compression and encryption so that it can be safely transferred and stored in the secondary location. After this, ssh is used to verify that a directory exists on the target machine or to force creation of that directory, then I use scp to transfer the file, and lastly another ssh to verify that the file exists on the other end.

This worked well in the past, however a change was made to the script to include a timestamp in the filename (in addition to the existing datestamp) and the : characters in that timestamp in the filename are causing the transfer to fail under the scp transfer and the following ssh to test the file's existence.

Currently, I am creating a DATE variable at the top of the script and setting it to the following: $(date --date=today +%FT%H:%M:%S%z), which produces a datestamp in the proper ISO 8601 format. This works fine to create the database dumps (postgres dumps created with pg_dump) as well as compression with gzip and encryption with openssl. Each of these generates a new file, with all names in the format HOSTNAME.DATABASE-dump-DATE.EXTENSION.

When the date was being provided to the filename without the time, and thus without the : character, it worked perfectly. The specific errors I receive now that there are : characters are on the following segments of code ($1 refers to the argument provided to the script, containing the name of the database to be backed up, and $DATE was defined above):

scp -P PORT "$(hostname).$1-dump-$DATE.backup.gz.aes" USER@HOST:"/backups/$(hostname)/"

This fails with the error ssh: could not resolve hostname HOSTNAME.DATABASE-dump-2015-07-31T13: Name or service not known.

ssh HOST -p PORT <<"HERE"
if test -f /backups/$(hostname)/$(hostname).$1-dump-$DATE.backup.gz.aes;
then echo "File exists";
else echo "Transfer failed";
fi
HERE

This fails with Pseudo-terminal will not be allocated because stdin is not a terminal. Using -t produces the same error, and using -t -t (from info found at stackoverflow that lead down the rabbit hole about Pseudo-terminal) does not allow the commands given after to flow through, which is probably something that I missed about what that actually does.

I would like to avoid using any additional software that would need to be installed on the machines (running Ubuntu 14.04 and latest Postgresql for DB). Please let me know if there is any way scp/ssh can be made to handle these files properly.

  • I notice that you use the date as variable $DATE. Just an idea. Can you try to put a \ before the :? – Marco Jul 31 '15 at 15:02
5
scp -P PORT "$(hostname).$1-dump-$DATE.backup.gz.aes" USER@HOST:"/backups/$(hostname)/"

Scp supports copying files from one remote host to another, and the syntax to do that is to specify both the source and destination using the "hostname:filename" syntax:

scp srchost:/src/file desthost:/dest/file

Your source file has a colon in its name, so scp is trying to parse it as the hostname and filename of a remote file.

You can prevent scp from treating the source filename argument as a remote filename by making sure the argument has a "/" before the first colon. Hostnames can't have "/" in them, and scp stops looking for colons in a command-line argument after the first "/". This ought to work:

scp -P PORT "./$(hostname).$1-dump-$DATE.backup.gz.aes" USER@HOST:"/backups/$(hostname)/"
             ^^-- Prepend a ref to the current directory
  • You're quite right, my approach (escaping the :) didn't work. I could have sworn I tested this yesterday and saw that it worked but I certainly can't reproduce it now. Thanks for pointing it out, I deleted my answer. – terdon Aug 1 '15 at 15:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.