Wait for multiple files to be finished downloading

Question

I have the following script which checks if there is a file in a folder and if the file is stable (this is for large video file I am receiving).

#!/bin/bash

cdate1=$(date +%Y%m%d-%T)
folder1="/path-to-folder"

cd $folder1

while file=$(ls "$folder1")
      [ -z "$file" ]
        do sleep 10
        done
            echo "There is a file in the folder at $cdate1"
            size1=$(stat -c '%s' "$file")
            echo "The size1 is $size1"
            sleep 30
            size2=$(stat -c '%s' "$file")
            echo "The size2 is $size2"
        if [ $size1 = $size2 ]
        then
            ls -l
            echo "Start converting"
        else 
            echo "Restart the script"
        fi

How to modify the script so it can check multiple files in the same folder and after all of them are stable to start the conversion script?

Answer 1

there are a number of problems in your script, starting with:

folder1="path-to-folder"
cd $folder1
while file=$(ls "$folder1") ...

this really does cd path-to-folder; ls path-to-folder; if path-to-folder is absolute (starting with a '/' as in your example) this might work, but it will not work whenever you use relative paths. it will also not work if path-to-folder contains spaces, as you should use quotes everywhere, e.g. cd "$folder1"

then doing a while file=$(ls ...) will do an infinite loop, as file will always be set to something (the contents of the directory).

the proper syntax would be for file in $(ls ...), which will stop working as soon as you have filenames with spaces (as the loop will run on foo and bar if you have a file named foo bar). check why you should never parse the output of ls. instead of using ls you could simply do for file in *.

finally, you could have files that are changing even if there size is not changing any more.

a good way to iterate over files is the find command; a good way to check whether something has changed is the mtime of a file.

the following function gives you a value for the last modification (mtime) of any file in the given directory

folder1="/path-to-folder"
find "${folder1}" -exec stat -c "%Y" \{\} \;    \
   | sort -n | tail -1

so your script could look like:

#!/bin/bash
dir="$1"
# check whether $dir exists
test -d "${dir}" || exit 1

last=0
current=1

while [ "$last" != "$current" ]; do
   last=$current
   current=$(find "${dir}" -exec stat -c "%Y" \{\} \; \
             | sort -n | tail -1)
   sleep 10
done
echo "directory is now stable..."

UPDATE

an even better approach would be to actively notify the receiver that a given file has been transmitted. a very simple solution would be to also copy an empty dummy file after the payload has been transmitted. e.g. for a file named foo.avi copy another file foo.avi.copyfinished; so you only need to check for the existance of foo.avi.copyfinished to see that foo.avi is ready.

while true; do
 for file_ready in *.copyfinished; do
   file=${file_ready%.copyfinished}
   if [ -e "${file}.converted" ]; then
      echo "skipping already converted file ${file}" 1>&2
   else
      touch "${file}.converted"
      do_convert "${file}"
   fi
 done
 sleep 1
done

this solution obviously requires the cooperation from the sending side.

Answer 2

It's easier to control processing on the sending side where the process knows if the transfer is completed. This way you won't pick up aborted transfers either.

Send the file to a temporary name or directory. When the send is finished move the file into the correct location

cp srcvideo.avi /folder1/srcvideo.tmp && mv /folder1/srcvideo.tmp /folder1/srcvideo.avi

Then your script doesn't need to do all the testing on sizes. It can just wait for the correct file.

cd /folder1 || exit 1
for file in *.avi; do
  echo "$file found"
  do_some_processing "$file"
done

You could achieve the same by using a seperate transfer directory without changing filenames if you wish.

Answer 3

Waiting for files to stop growing is a terrible way of detecting whether a download is finished. If the download pauses for too long due to a network glitch, your script will fire up.

The best way to do something upon the completion of a download is to either instruct the download program to run your script when the download is complete and successful, or to wait until the download program exits and then run your script if the download was successful. Any decent download program will allow at least one of these options.

If you are stuck with some inferior downloading method, use a notification facility to react when the file has finished downloading, instead of manually watching the file. On Linux, the notification facility is inotify. Depending on how the download program works, either react to a file closure (if the download program directly writes to the final file) or to a rename (if the download program first writes to a temporary file then renames it into place).

You can use the shell tool inotifywait to react when a file event occurs. Here's an example that converts every renamed file.

cd /path/to/directory
inotifywait -m -e moved_to --format=%f . |
while IFS= read -r filename; do
  conversion-program "$filename"
done

Answer 4

If you want to launch your script before you know if all the files are there, the first thing is to check if the number of file is constant. You could do something like, using listfile = $(ls "$folder")

[ $( echo $listfile | wc -w) != $(ls "$folder" | wc -w) ]

in a while loop; till the thing are not equal, your reset $listfile to $(ls "$folder"). Once it is done, you can simply do a doloop on $listfile where you do exactly what you were doing on one file; once it is done, you can launch the conversion.