Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have shell script which waits for some files to arrive from remote machine, when it arrives, it cats them to a new file. For that I am using a while loop,like following:

while true
do

    if [ $(find ../Test_Data/local_enc* | wc -l) -eq 2 ]
    then
            break
    else
            sleep 0.001

    fi
done
cat ../Test_Data/local_enc* > ../Test_Data/All_Enc_Coords.txt

The problem is All_Enc_Coords file is written sometimes, and sometimes not written. I think this is because, find function returns some value, even when the file has no data in it.

How to ensure that the files have been written successfully?? and how to specify it in the while loop. That means, I want all the files to be written completely and then cat them?.. I used sleep 1 before cat and found that the file is written completely, but is there any way round to check this??

share|improve this question
4  
"Written completely" is an application-level concept. You can't guarantee from within your script that the files are "complete" unless whatever is producing that data gives you some form of signal that it's done writing. All the checks you can do from "outside" are inherently racy. –  Mat Sep 29 '12 at 8:14

3 Answers 3

up vote 0 down vote accepted

You can use a test case.

while [[ ! -e ../Test_Data/All_Enc_Coords.txt ]]; do
  if [ $(find ../Test_Data/local_enc* | wc -l) -eq 2 ]; then
    cat ../Test_Data/local_enc* > ../Test_Data/All_Enc_Coords.txt
  else
    sleep 0.001
  fi
done

As per the comments, if you were checking for the file containing data before writing, you could use:

while [[ ! -s ../Test_Data/All_Enc_Coords.txt ]]; do
  if [ $(find ../Test_Data/local_enc* | wc -l) -eq 2 ]; then
    cat ../Test_Data/local_enc* > ../Test_Data/All_Enc_Coords.txt
  else
    sleep 0.001
  fi
done
share|improve this answer
1  
I don't think that test will help the asker. -e tests if the file exists, not whether it contains data. –  Mat Sep 29 '12 at 8:07
    
@Mat He pointed out that it is creating a new file. –  Sly Sep 29 '12 at 8:11
    
what does -s test?? –  MiNdFrEaK Sep 29 '12 at 8:42
    
@MiNdFrEaK: read man test for the answer to that & all the other options. –  Mat Sep 29 '12 at 9:50

Can you get the remote machine to create a file just before it starts uploading the files, and then delete it afterwards?

e.g. using ssh (similar could be done with ftp or a HTTP PUT):

ssh yourhost touch ../Test_Data/upload-in-progress
scp local_enc* yourhost:../Test_Data/
ssh yourhost rm ../Test_Data/upload-in-progress

Then all your script has to do is wait for the upload-in-progress file to disappear. This could be done with a sleep loop, or perhaps using inotifywait from the inotify-tools package.

NOTE: If the remote host dies or its script is killed before it completes the upload, it will leave a stale upload-in-progress file around. IMO this is a much smaller problem than the risk of race-conditions from trying to guess when the upload has finished (as all solutions running on the destination machine alone will be prone to)

I initially thought of using lsof | grep local_enc | wc -l, but that's just as prone to races as your find .. | wc -l.

As is using inotify or similar to get notified of changes to the Test_Data directory - you can tell when files are created/altered in that directory but that doesn't tell you when an upload session has completed...however inotify in combination with a semaphore file would work. inotify to wait for the Test_Data dir to be changed, then inotify to sleep until upload-in-progress has been deleted.

Also, if the upload script on the remote host runs from cron the stale upload-in-progress file will fix itself on the next run. Alternatively, your script could be written to assume that any upload-in-progress file older than X minutes is stale and should be deleted/ignored (inotifywait has a -t or --timeout option which would be useful here), but network delays or temporary outages could cause you problems here.

share|improve this answer

If cat doesn't print any errors, it was succeed. If you see no data in All_Enc_Coords.txt, check if local_enc* is empty as well.

You don't need to monitor file writes, if cat quits, the writing is finished; If this is running in another script, use inotify to listen for the WRITE_CLOSE signal.

share|improve this answer
    
no local_enc* is not empty.. –  MiNdFrEaK Sep 29 '12 at 8:36

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.