I do simulations on large supercomputers, and the files I produce are on the order of 5 GB to 1 TB. But I frequently like to monitor the progress of the simulation using a visualization program on my laptop, so I'll transfer the log/data files while the simulation is still running. When the simulation is over, I have to SCP the whole file again. Is there a way to update just the new part that was added?
You can use
rsync with the
--append-verify option. See the rsync(1) man page for details.
Note: even without
rsync will be much more efficient than
scp (but since your data are really big, you may need one of these options anyway).