I do simulations on large supercomputers, and the files I produce are on the order of 5 GB to 1 TB. But I frequently like to monitor the progress of the simulation using a visualization program on my laptop, so I'll transfer the log/data files while the simulation is still running. When the simulation is over, I have to SCP the whole file again. Is there a way to update just the new part that was added?

  • Consider using the Folding@Home or Seti@Home Projects to run/watch your simulation. – eyoung100 Nov 7 '14 at 17:36
  • 1
    That won't work; my simulations require direct interprocessor communication (I'm using LAMMPS). – Nick Nov 7 '14 at 17:52
  • My direction was this, but I can't tell you how to do it. Trade your file size in for smaller file sizes, and then chain the files together into work units. Your current work unit is TOO BIG to be productive. The normal software to use wd be git or tortiose, but even those monitor files, so thats out... – eyoung100 Nov 7 '14 at 18:03
  • sftp is better to use but it will not solve your problem, you can transfer only updated and append it to file there. wt i suggest. transfer the updated data using a new file and do ssh and then cat new_log >> old_log – Hackaholic Nov 7 '14 at 19:02

You can use rsync with the --append or --append-verify option. See the rsync(1) man page for details.

Note: even without --append or --append-verify, rsync will be much more efficient than scp (but since your data are really big, you may need one of these options anyway).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.