How to know if a file has been written completely?

Question

I have shell script which waits for some files to arrive from remote machine, when it arrives, it cats them to a new file. For that I am using a while loop,like following:

while true
do

    if [ $(find ../Test_Data/local_enc* | wc -l) -eq 2 ]
    then
            break
    else
            sleep 0.001

    fi
done
cat ../Test_Data/local_enc* > ../Test_Data/All_Enc_Coords.txt

The problem is All_Enc_Coords file is written sometimes, and sometimes not written. I think this is because, find function returns some value, even when the file has no data in it.

How to ensure that the files have been written successfully?? and how to specify it in the while loop. That means, I want all the files to be written completely and then cat them?.. I used sleep 1 before cat and found that the file is written completely, but is there any way round to check this??

Answer 1

Can you get the remote machine to create a file just before it starts uploading the files, and then delete it afterwards?

e.g. using ssh (similar could be done with ftp or a HTTP PUT):

ssh yourhost touch ../Test_Data/upload-in-progress
scp local_enc* yourhost:../Test_Data/
ssh yourhost rm ../Test_Data/upload-in-progress

Then all your script has to do is wait for the upload-in-progress file to disappear. This could be done with a sleep loop, or perhaps using inotifywait from the inotify-tools package.

NOTE: If the remote host dies or its script is killed before it completes the upload, it will leave a stale upload-in-progress file around. IMO this is a much smaller problem than the risk of race-conditions from trying to guess when the upload has finished (as all solutions running on the destination machine alone will be prone to)

I initially thought of using lsof | grep local_enc | wc -l, but that's just as prone to races as your find .. | wc -l.

As is using inotify or similar to get notified of changes to the Test_Data directory - you can tell when files are created/altered in that directory but that doesn't tell you when an upload session has completed...however inotify in combination with a semaphore file would work. inotify to wait for the Test_Data dir to be changed, then inotify to sleep until upload-in-progress has been deleted.

Also, if the upload script on the remote host runs from cron the stale upload-in-progress file will fix itself on the next run. Alternatively, your script could be written to assume that any upload-in-progress file older than X minutes is stale and should be deleted/ignored (inotifywait has a -t or --timeout option which would be useful here), but network delays or temporary outages could cause you problems here.

Answer 2

There are two ways you can achieve it.

1. Check if file has not been touched for 2 or 3 minutes after it is written. That way, you can say that file is fully written or not. To check if the file has been written a minimum of 3 minutes before:

   find /testfolder/filename.* -type f -mmin +3
   

   You can use a for loop if you have more than one file:

   for f in $(find /testfolder/filename.* -type f -mmin +3)
   do
     mv filename.* to destination
   cat ../Test_Data/local_enc* > ../Test_Data/All_Enc_Coords.txt
   done
   

2. If you have trailer in the file, then you can read the trailer record and then decide when to catalog the file.

Answer 3

You can use a test case.

while [[ ! -e ../Test_Data/All_Enc_Coords.txt ]]; do
  if [ $(find ../Test_Data/local_enc* | wc -l) -eq 2 ]; then
    cat ../Test_Data/local_enc* > ../Test_Data/All_Enc_Coords.txt
  else
    sleep 0.001
  fi
done

As per the comments, if you were checking for the file containing data before writing, you could use:

while [[ ! -s ../Test_Data/All_Enc_Coords.txt ]]; do
  if [ $(find ../Test_Data/local_enc* | wc -l) -eq 2 ]; then
    cat ../Test_Data/local_enc* > ../Test_Data/All_Enc_Coords.txt
  else
    sleep 0.001
  fi
done

Answer 4

If cat doesn't print any errors, it was succeed. If you see no data in All_Enc_Coords.txt, check if local_enc* is empty as well.

You don't need to monitor file writes, if cat quits, the writing is finished; If this is running in another script, use inotify to listen for the WRITE_CLOSE signal.

Answer 5

I am using below code to check if data is still coming to file. If not then it is written completely. Loop exits and next processing starts.

for (( ; ; ))
do
    bfr=$(stat -c%s "$f")
    sleep 0.5
    aftr=$(stat -c%s "$f")
    if [ $bfr -eq $aftr ];
    then
        break;
    fi
done

Answer 6

while true
do
    echo "Checking for file : $file_with_full_path"
    if [ -f "$file_with_full_path" ]; then
            echo "**File arrived**"
        for (( ; ; )); do
                echo "getting the file size"
            bfr=$(stat -c%s "$file_with_full_path")
                echo "File size at present: $bfr"
            sleep 30 # the sleep limit can be adjusted
            aftr=$(stat -c%s "$file_with_full_path")
            if [ $? -ne 0 ];then echo "File not present, looping again";break;fi #error handling, incase if file got removed or renamed after the transfer
                echo "File size at present: $aftr"
            if [ $bfr -eq $aftr ];then
                    echo "**File arrived and file size matched exiting the loop, before vs after size : $bfr vs $aftr**"
                break 
            fi
                echo "File arrived but file size doesn't match looping again: Before vs after : $bfr vs $aftr"
        done
        if [ -f "$file_with_full_path" ]; then  break; fi #double break to exit the loop
    else
            echo "File not available Sleeping and looping again"
        sleep 30 # wait limit can be adjusted
        continue
    fi
done

Answer 7

lsof command is made for this. Its lists all the processes which have opened the file.