I have a situation that needs a solution like the original poster asked. I am recording a hockey game on my computer in one location and I'd like to watch it on my TV at another location. The link between the two locations allows the copy to go at about 1.3Mb/s and the recording video is about 1.5Mb/s. So, I want to copy the file as it starts recording. This way my 3 hour game will copy in about 3.5 hours. So, I copy it as it starts recording and I can start watching it 30 minutes after it starts. Then I can watch it with no interruptions, almost in real time. That is, as long as I can get it to copy as its writing the new file. The problem with tools like rsync and scp is that they look at the size of the file when you initiate the copy and once it copies that amount of data, it quits; even if the file has grown by more than double during that copy. And, if, I'm just using rsync in a loop to copy it once it stops, when the next rsync finishes it rebuilds the target file and that kills my video player and I have to restart watching it and fast forward to wherever I was in the program when it suddenly killed it. I wanted a better solution and I haven't been able to find one, so I pieced together this instead:
dd if=2031_20160514030000.mpg | pv --size 4653819304 | ssh -C -c arcfour,blowfish-cbc -p 5555 myserver.com 'dd of=/media/TV/2031_20160514030000.mpg'
So what does this do?
First, I use dd to copy the file as it grows. Since the file grows faster than dd can send it over the network, dd never catches up to the end of the file.
Next, I pipe it to "pipe viewer (pv)" and I give it an estimate on how big the file is going to be based on how big these files usually are. This isn't necessary, but I like to see a progress meter.
Then, I pipe the stream to my ssh connection. The ssh connection uses -C for compression (to reduce the network bandwidth and try to speed it up), -c arcfour,blowfish-cbc for the least expensive encryption (again to speed things up a bit), the -p is for my firewall port I'm using at the destination, and the ssh finally runs the dd command on the target to recreate the file as it receives it. I'm happy to say, this solution works great. I can watch the hockey game while the file is being created and copied with only a short delay. Hope this helps other people looking for a similar setup. I couldn't find anything that worked this well, even though I found a lot of people describing questions just like the one here and what I wanted, so I thought I should share this in the hopes it helps someone else one day.