Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Here is the situation:

  1. I am uploading a large file from client A to a server using sftp.
  2. I also need to download this file from the server to client B over ssh.

What I would like to do is start the transfer from the server to client B when the upload is still happening from client A.

What is the best method/tool to get this done?

UPDATE:

The answers so far are interesting--I'll be sure to read and test them all. Bonus points for answers that don't depend on controlling how Client A is uploading the file. (ie. the only thing we know from client A is that the file is being written to a known filename.)

share|improve this question
    
Ooo, good question. This is certainly possible, but I'm not aware of anything that implements it –  Michael Mrozek Sep 30 '10 at 19:23
add comment

5 Answers

up vote 8 down vote accepted

For a single file instead of using SFTP you could pipe the file over ssh using cat or pv at the sending side and using tee on the middle server to both send the data to a file there and send a copy over the another ssh link the other side of which just writes the data to a file. The exact voodoo required I'll leave as an exercise for the reader, as I've not got time to play right now (sorry). This method would only work if the second destination is publicly accessible via SSH which may not be the case as you describe it as a client machine.

Another approach, which is less "run and wait" but may otherwise be easier, it to use rsync between the server and client B. The first time you run this it may get a partial copy of the data, but you can just re-run it to get more data afterwards (with one final run once the Client1->Server transfer is complete). This will only work if the server puts the data direct into the right file-name during the SFTP transfer (sometimes you will see the data going into a temporary file which is then renamed once the file is completely transferred - this is done to make the file update more atomic but will render the rsync idea unusable). You could also use rsync for the C1->S transfer instead of scp (if you use the --inplace option to avoid the problem mentioned above) - using rsync would also give you protection against needing to resend everything if the C1->Server connection experiences problems during a large transfer (I tend to use rsync --inplace -a --progress <source> <dest> instead of scp/sftp when rsync is available, for this "transfer resume" behaviour).

To summarise the above, running:

rsync --inplace -a --progress <source> user@server:/<destination_file_or_folder>

on client1 then running

rsync --inplace -a --progress user@server:/<destination_file_or_folder> <destination_on_cli2>

on client2 repeatedly until the first transfer is complete (then running once more to make sure you've got everything). rsync is very good at only transferring the absolute minimum it needs to update a location instead of transferring the whole lot each time. For paranoia you might want to add the --checksum option to the rsync commands (which will take much more CPU time for large files but won't result in significantly more data being transfered unless it is needed) and for speed the --compress option will help if the data you are transferring is not already in a compressed format.

share|improve this answer
    
Thanks, in the end I went with the rsync approach. –  Steven D Oct 25 '10 at 19:16
add comment

I can't try it at the moment, so this might well fail: My idea is this: Mount the directory where the file is arriving in client B, e.g. with sshfs to /mnt/server in the filesystem of client b. Then

tail -c +0 -f /mnt/server/thefileinquestion > ~/finalfile
share|improve this answer
    
/usr/bin/tail: cannot open `+0' for reading: No such file or directory - coreutils 7.4 –  maxschlepzig Oct 1 '10 at 6:26
    
Sorry, there was a -c missing. I fixed it in the answer above. –  fschmitt Oct 1 '10 at 7:22
    
ok, a problem I see with this is that the command does not terminate (-f -> follow ...). One has to issue a sigQUIT or something like that, when you are sure that thefileinquestion is completely written. Btw, depending on your tail version and fs, tail internally does polling of the file (e.g. every second). –  maxschlepzig Oct 1 '10 at 7:56
add comment

I think this should work:

user@clientA:~$ cat file | ssh server "cat > dest"

and then

user@clientB:~$ ssh server "tail +0 -f dest" > file

Add the pv command if you want to see your throughput.

share|improve this answer
add comment

You could use a fifo for it. For simplicity first without ssh only involving two xterms:

At xterm A:

$ mkfifo fif
$ cat test.tar.gz | tee copy.tar.gz > fif

At xterm B:

$ cat fif > dest.tar.gz
$ cmp test.tar.gz dest.tar.gz
$ echo $?
0
$ cmp test.tar.gz copy.tar.gz
$ echo $?
0

With ssh it should be something along these lines - perhaps you have to disable the escape-character in ssh (-e none):

client A:

 $ ssh server mkfifo fif
 $ cat src.tar.gz | ssh "tee fif > copy.tar.gz"

client B:

 $ ssh server cat fif > dest.tar.gz
share|improve this answer
add comment

I'm not sure that the tail -f method works ( though it probably does if the file is text ). The reason is that I don't know how tail -f, and sftp transfer and rely on meta information.

If sftp transfers the meta information first and tail -f relies on the meta information to tell it that there is no more file, then tail may bad the end with EOFs or nulls.

If you do not care about the path of upload ie Computer 1 uploads to computer 2 uploads to computer 3, then you might try top use bittorent instead of sftp. It seems that is what it was designed for.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.