1

We need to do a once-only archive copying of users' home folders to an archive server (pending final deletion) when they leave, in case they later discover that they may still require some of their files (although we do of course very strongly encourage them to take their own backup of everything they might still need before they go).

We had been using scp for this, but have now got inadvertently snared by a former user who had installed some software which had created an unusual symlink structure in one of their folders, which seemed to result in scp looking ever upwards and then trying to copy rather more than was expected, before being stopped.

Unfortunately, it turns out that scp seems to always follow symlinks and does not appear to have any option to prevent this.

I am looking for an alternative way to backup a user folder that avoids this problem (and ideally is no more complicated than it absolutely needs to be).

tar could be a possibility, but I am slightly concerned that the creation of a tarball locally before copying it to the archive server could use a not insignificant amount of storage space, and might pose some difficulties in the event that our fileserver becomes rather more full at some point in the future.

Another possibility might be to use rsync, but this seems possibly over-the-top for a once-only file transfer, and I know from previous experience that tuning rsync's own options can sometimes be fiddly in itself.

Can anyone suggest a reliable and simple alternative to scp for this?

Improve this question

2 Answers 2

Reset to default
2

If you like tar except for the temp file, this is easy: don't use a temp file. Use a pipe.

cd /home ; tar cf - user | gzip | ssh archivehost -l archiveuser 'cat > user.archived.tar.gz'

Substitute xz or whatever you prefer for gzip. Or move it over to the other side of the connection, if saving CPU cycles on the main server is more important than saving network bandwidth (and CPU on the archive server)

cd /home ; tar cf - user | ssh archivehost -l archiveuser 'gzip > user.archived.tar.gz'

You could stick a gpg in there too. Have a key pair that's just for these archives, encrypt with the public key when storing, use the private key when you need to recover something.

More details as requested:

I intend user to be the user whose home directory /home/user you are archiving. archivehost is the server where you're going to store the archives, and archiveuser is an account on the archive server that will own the archives.

tar cf - user means "create a tar archive of user and write it to stdout". The -c is "create", and -f - is "use stdin/stdout as the file". It will probably work with just tar c user since -f - is likely to be the default, but originally the default action of tar was to read or write a tape device. Using an explicit -f - may just be a sign that I'm old.

The tar z flag would be fine, except then I couldn't show how to move it to the other side of the ssh. (Also, connecting gzip and tar with an explicit pipe is one of those "old people" things - tar didn't always have that option.) Plus I can substitute bzip2, lzop, xz -3v, or any other compression program without needing to remember the corresponding tar options.

I never heard of --checkpoint before, so you'll just have to rely on your own tests for that one.

Improve this answer
3
  • Thanks, that's ingenious! I did not know that you could pipe just data (rather than commands) across the network through ssh. Could I just ask a couple of things for clarification? What does the "tar cf - user" mean? I assume that "user" is the name of the folder to be tarred up, but what does the "-" mean? Does that mean to divert the output to a pipe (since the -f option usually requires a tar file name)? It's not an option I was aware of. Is there a particular reason for the gzip stage as a separate command (rather than using -z)? Would it not work over the pipeline(s) otherwise?
    – dave559
    Commented Jun 23, 2017 at 9:34
  • The tar through ssh-pipe solution seems to work well. I am wondering if I can also use the "--checkpoint" option to tar to show progress (where there is a lot of data to transfer (the archiving script will be run manually)). I tested with a smallish test folder, and it looks as though this might cause tar to create all(?) of the datastream locally first, and only then starting ssh to transfer the data (I see all(?) checkpoint output and seem to be prompted for the password for my test server only after that), which might be a space issue depending on where the data is (stored?) at that time.
    – dave559
    Commented Jun 23, 2017 at 12:00
  • Answering my own query after further testing: yes, it does seem that I can use the "--checkpoint" option successfully; after trying with a larger amount of data it appears that perhaps the pipe requires some "priming" before data starts to pass through it from the source computer to the destination computer.
    – dave559
    Commented Jun 23, 2017 at 12:17
1

Rsync is a reasonable choice. There's nothing wrong in using rsync for a one-time transfer.

Tar is also a reasonable choice. You don't need to store the archive locally, you can directly pipe it to SSH.

Another possibility is to mount the server directory over SSHFS. However, beware that SSHFS won't preserve all metadata such as permissions. (Tar and rsync might not preserve all metadata either but they can do a better job — pass --acls --xattrs to GNU tar, pass --acls --xattrs to rsync.)

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .