4

I have a directory on a remote server remote1, mounted via sshfs. I am installing new versions of some of the files in the remote directory, which are quite large (~50M).

I am currently using

rsync -a --in-place local/file.so remote1/file.so~new

to copy the files into the remote directory, and then

rsync -a --in-place remote1/file.so~new remote1/file.so.

The second rsync is slower than I would like and I suspect that it is downloading the file data via ssh before re-uploading them, rather than copying on the remote system.

I would prefer to keep the file.so~new file as a backup (I am switching between the new and original file), which is why I am not using mv.

I don't have an rsync program on the remote host, which is why I am not using a standard rsync command, but I am using rsync to avoid recopying when the file is already in place.

Are there command line options that will improve the copy speed? Would I be better off using scp or sftp? If so, how would I check that the file has not changed (time and size), so that I don't copy unnecessarily if the file is already in place.

My latest improvement is to upload two new copies and then use mv to rename one of them. Is there no better way?

Improve this question
6
  • the second rsync could be done much faster by using ssh to run mv on the remote machine e.g. ssh user@remote "mv /mydir/file.so~new /mydir/file.so"
    – gogoud
    Jan 19, 2016 at 20:35
  • @gogoud I want to keep a backup, which is why I'm not using mv. Please see edited question.
    – Mark Perryman
    Jan 20, 2016 at 14:02
  • Don't use sshfs if you want fast file copies.
    – mikeserv
    Jan 20, 2016 at 17:17
  • I'm looking for a good compromise, not optimal speed. Maybe I have it as good as it gets.
    – Mark Perryman
    Jan 21, 2016 at 9:13
  • 1
    so if you want to keep the copy, why not: ssh user@remote "cp -a /mydir/file.so~new /mydir/file.so". This just uses cp on the remote machine and you only have to upload the file once. For the first step you might use scp command instead of using sshfs, it might be faster (and I'm sure will not be slower).
    – gogoud
    Jan 21, 2016 at 13:03

1 Answer 1

Reset to default
4

When rsync believes it's accessing both source and destination on the same host it does not use its incremental transfer. Instead, it simply copies the entire file.

This is the situation in your case. As a result, your second command is copying the file.so~new across your ssh fuse link to the local host, and then immediately copying it back again.

Is there anything wrong with using this instead?

mv remote1/file.so~new remote1/file.so

To get the fastest speed you should not try to run rsync over sshfs but instead allow it to talk to the remote server directly (over ssh). You could then use the simple variant:

rsync -av local/file.so remote:path/to/file.so

The new file won't be installed until it has been completely copied, and you have the advantage of being able to use the old library as a base for the new one.

Improve this answer
2
  • Thanks. I have edited the question to explain why I did not use either of your suggestions. I want to keep a backup copy of the new file as backup (hence wanting to copy rather than move), and the simple rsync command requires rsync on the remote server, which I do not have.
    – Mark Perryman
    Jan 20, 2016 at 14:04
  • @MarkPerryman without rsync on the remote server, all it implements is cp
    – roaima
    Jan 20, 2016 at 16:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.