I have a large compute job running on a remote machine that generates ~40 data files every ~20 minutes. I would like to pull the generated files from the remote machine to my local machine as soon as they are generated, and immediately delete them from the remote machine.

I've gotten part of the way there using rsync --remove-source files user@remote:~/datadir/* ./localdir. However, this does not run rsync "live" i.e. if new files are added to datadir I need to re-run rsync.

To my understanding, rsync first creates a list of files to copy, then goes through the list one by one. I am wondering, is there a way to update the list as new files are added to datadir, or some other way to move files from the remote machine to local as soon as they are generated?


Well, this is all kind of like when you go to your doctor and say "Doc! It hurts when I do this!" and he says "So don't do that! Problem solved!".

rsync is for synchronizing directories but, from your description, you don't want to do that. You want to "unsync" two directories: you want the files to be in one, but not both of them.

I take it that you can't modify the code generating the files on the other system since, if you could, you would simply have it run ftp or rcp or curl or some such as soon as the files are finished to push them over to the target machine from the source machine and then delete them.

So working only from the target machine you're best off to just run a periodic job to sign on to the remote machine and copy and delete everything in ~/datadir. It'll save rsync's overhead of comparing the two directories: this being wasted effort since you don't care about the contents of ./localdir - it's always going to get the contents of ~/datadir pulled into it. Using rcp or scp is the simplest but if the only access you have on the remote system is rsync then run that in a cron job.

This will cause a delay between file creation and transport of some few minutes depending on the periodicity of your job since, as you note, rsync doesn't run live. If you need immediate transport you'd have to run a file sharing server such as amule on the other machine but that's a lot of complexity and overhead to save a few minutes and you'd still have to sign on every so often to delete the files: something no file sharing utility is going to do for you.

In all of this there is the lurking gremlin of How do you know you've got complete and uncorrupted files waiting for you on the other end?

If you're just taking things from the directory in which the files are created then any number of things can result in your taking some fraction of a complete file over to your target machine. You could, for example, start the copy (or rsync or whatever) while the file is being written out by the code on the source machine. Or the creating program could just fall over due to some hardware problem, such as a full disk, part way through creation.

So when doing this sort of thing I always have separate creation and transportation directories and then mv (not cp) files after successful creation from the creation to the transportation directory. I'm very paranoid about file corruption too so I always cook my own digest/checksum/manifest file for the source files as well on top of all the automatic low level stuff that guards against corruption.

  • "can't modify the code generating the files" - I can modify the code with no problem, but what I can't do is ssh from the remote machine into my local machine either because of firewall settings or because I don't know what I'm doing. If I can ssh into the remote machine from my local does that guarantee I can go the other way? – dkv May 12 '17 at 16:09
  • Making a transportation directory is a great tip, I was actually concerned about the very problem you're describing about copying as it's being written. I don't have the technical competence to write my own corruption-proofing, but doesn't rsync run checksums on every block it writes automatically? – dkv May 12 '17 at 16:11
  • @dkv Well no, there are many reasons why you might not be able to ssh from B to A even if you can ssh from A to B but that shouldn't matter. If you have ssh access to the shell on B then you already have a link that allows file manipulation on B and data transfers from B to A. It would be weird if neither scp nor sftp, both of which are built on top of ssh, were available to easily pull data back through your ssh connection. See their man pages for usage. I actually prefer lftp (maybe not on your B system) due to its better automation. Use cron to run things periodically. – Nadreck May 12 '17 at 16:46
  • @dkv All sorts of things run error checks on all typical file transfers including, as you say, rsync. I often deal in situations using very old equipment running in rotten environments or where active sabotage is a consideration so I'm, as I mentioned, paranoid and often lard another layer on top of those checks. Probably not a consideration in your case. – Nadreck May 12 '17 at 16:53
  • 1
    @dkv To be precise, what I meant was that you can't modify your code to get it to push from B to A. If B to A ssh isn't available then you can't. Getting it available is a whole other question possibly involving admin access on B or control of firewalls. The best you can do towards immediate transfer is to send some sort of signal to A ( email, UDP blip) to start the pull. These may or may not be available due to the same problems that ssh has going from B to A. You have to ask yourself the effort it is worth to avoid the few minutes delay involved in a cyclically running pull. – Nadreck May 12 '17 at 17:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.