Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have an application running that is generating a large (~200GB) output file, and takes about 35 hours to run (currently I'm about 12 hours in). The application just opens the file once then keeps it open as it is writing until it is complete; the application also does a lot of random access writes to the file (i.e. not sequential writes).

Right now the file is being saved to my local hard drive but I just decided that when it's done, I'm going to move it to a different device instead (a network drive, NTFS mounted via CIFS).

To save time instead of moving the file later, is there some way I can suspend the program and somehow move the current partially complete file to the other device, do some tricks, and resume the program so it is now using the new location?

I'm pretty much positive that the answer is no but I thought I'd ask, sometimes there are surprising tricks out there...

share|improve this question
    
Is the remote drive mounted using NFS? –  Joshua Huber Jun 11 at 1:14
    
@JoshuaHuber No, it's CIFS. –  Jason C Jun 11 at 18:03

3 Answers 3

up vote 3 down vote accepted

Posting another solution, since file is being written randomly, breaks my tail idea. Thinking rsync might be promising here, since the rsync can operate using a delta transfer algorithm, saving transfer time by only sending the changed parts of a file. If you run rsync on two local files, it will default to --whole-file mode, which is not what you want.

Suggesting

rsync -av --inplace --no-whole-file /your/local/file.dat /your/remote/file.dat

... or maybe (if the CIFS mount doesn't agree with delta transfer) use pure rsync:

rsync -av --inplace --no-whole-file /your/local/file.dat remoteserver:/your/directory/file.dat

So you would run this multiple times while your 200 GB file is filling up. Each time you run it, it updates the remote file incrementally. This should even work when the source file is being updated randomly. Maybe you could run this every 15 minutes. Then when your pid finishes, you would run it once more, and it would just be a quick incremental delta.

share|improve this answer
    
Brilliant! This accomplishes exactly what I want with minimal wasted time (guess I fell into a classic XY problem here). I have it in a script that runs in 10 minute delays (10 minutes between rsyncs; not 10 between starts). It seems to work fine via the plain CIFS mount path. The other answers here are good too for more specialized situations. Thanks! –  Jason C Jun 11 at 19:27

Assuming the network drive is mounted (NFS or whatever),

tail -f -c1000000000000 /your/local/file.dat > /mnt/nfs/drive/file.dat

This will start copying your output file to the network drive and keep it synchronized. When your program finishes, you should see the same 200GB file on both local and network fs. Maybe do a md5sum on both to verify.

Some unix flavors might also support also running tail with --pid={Your pid here} which will automatically stop tailing when your pid finishes.

share|improve this answer
    
I should note that this will only work if your process writes the file sequentially (usually the case). –  Joshua Huber Jun 11 at 1:54
    
This is a great time-saving idea. As it turns out, though, the file is written randomly, not sequentially. I will add that info to the question. Still, for sequential files, this is a much better way than what I was thinking. –  Jason C Jun 11 at 18:06
    
Dang, random access writing <sound of wrench in machine grinding to stop>. Posting another answer below that came to me after I thought of tail. –  Joshua Huber Jun 11 at 19:05

You could try:

  1. Suspend your application strg+z
  2. Jump to the directory where the file is generated cd CURRENT_CREATION_DIR
  3. Move the generated file to the new location mv FILE /path/to/new/location/
  4. Create a symlink to the file in the new location ln -s /path/to/new/location/FILE
  5. Continue your application fg [ENTER]

CAUTION:
Maybe you should start a second instance of your program, generating a testfile and test the above steps on the testfile first. I don't know if your application is fine with a symlink and just goes on. It is possible that the program fails.

share|improve this answer
    
This will work only if the process will close its filehandle on suspend and re-open it when foregrounded. As a rule, I wouldn't say that's likely. –  Jenny D Jun 11 at 7:48
    
You're right, but it's worth a try. And it's probably even faster than finding out if the application is closing the filehandle on suspend. –  xx4h Jun 11 at 7:52
    
Thanks, this is a good general trick, but I know for a fact that the application just opens the file once (when it starts) then keeps it open as it is writing until it is complete. –  Jason C Jun 11 at 18:04
1  
By the way, if you know the process closes the file periodically, you could use e.g. lsof during suspend to confirm that you interrupted it during a window where the file was closed. –  Jason C Jun 11 at 18:11
    
Right...didn't come to my mind... :> –  xx4h Jun 11 at 18:14

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.