4

I have the following script which checks if there is a file in a folder and if the file is stable (this is for large video file I am receiving).

#!/bin/bash

cdate1=$(date +%Y%m%d-%T)
folder1="/path-to-folder"

cd $folder1

while file=$(ls "$folder1")
      [ -z "$file" ]
        do sleep 10
        done
            echo "There is a file in the folder at $cdate1"
            size1=$(stat -c '%s' "$file")
            echo "The size1 is $size1"
            sleep 30
            size2=$(stat -c '%s' "$file")
            echo "The size2 is $size2"
        if [ $size1 = $size2 ]
        then
            ls -l
            echo "Start converting"
        else 
            echo "Restart the script"
        fi

How to modify the script so it can check multiple files in the same folder and after all of them are stable to start the conversion script?

10
  • 1
    You really shouldn't parse ls...
    – jasonwryan
    Sep 18, 2013 at 7:24
  • @jasonwryan Could you be more specific, please.
    – ispasov
    Sep 18, 2013 at 7:27
  • 1
    @ispasov $size1 = $size2 it will always be same , what you want to compare between file, in which basis ? can you please explain what you want to achieve Sep 18, 2013 at 7:39
  • @RahulPatil I am receiving large video files in that folder. So I compare the size because I don't want the conversion script to start during the file transfer. The transfer lasts couple of minutes.
    – ispasov
    Sep 18, 2013 at 7:43
  • size compare with each other or what ? Sep 18, 2013 at 7:45

4 Answers 4

4

there are a number of problems in your script, starting with:

folder1="path-to-folder"
cd $folder1
while file=$(ls "$folder1") ...

this really does cd path-to-folder; ls path-to-folder; if path-to-folder is absolute (starting with a '/' as in your example) this might work, but it will not work whenever you use relative paths. it will also not work if path-to-folder contains spaces, as you should use quotes everywhere, e.g. cd "$folder1"

then doing a while file=$(ls ...) will do an infinite loop, as file will always be set to something (the contents of the directory).

the proper syntax would be for file in $(ls ...), which will stop working as soon as you have filenames with spaces (as the loop will run on foo and bar if you have a file named foo bar). check why you should never parse the output of ls. instead of using ls you could simply do for file in *.

finally, you could have files that are changing even if there size is not changing any more.

a good way to iterate over files is the find command; a good way to check whether something has changed is the mtime of a file.

the following function gives you a value for the last modification (mtime) of any file in the given directory

folder1="/path-to-folder"
find "${folder1}" -exec stat -c "%Y" \{\} \;    \
   | sort -n | tail -1

so your script could look like:

#!/bin/bash
dir="$1"
# check whether $dir exists
test -d "${dir}" || exit 1

last=0
current=1

while [ "$last" != "$current" ]; do
   last=$current
   current=$(find "${dir}" -exec stat -c "%Y" \{\} \; \
             | sort -n | tail -1)
   sleep 10
done
echo "directory is now stable..."

UPDATE

an even better approach would be to actively notify the receiver that a given file has been transmitted. a very simple solution would be to also copy an empty dummy file after the payload has been transmitted. e.g. for a file named foo.avi copy another file foo.avi.copyfinished; so you only need to check for the existance of foo.avi.copyfinished to see that foo.avi is ready.

while true; do
 for file_ready in *.copyfinished; do
   file=${file_ready%.copyfinished}
   if [ -e "${file}.converted" ]; then
      echo "skipping already converted file ${file}" 1>&2
   else
      touch "${file}.converted"
      do_convert "${file}"
   fi
 done
 sleep 1
done

this solution obviously requires the cooperation from the sending side.

6
  • Don't parse the output of ls. Also, you're confusing for and while in the middle — the idiom to loop over the files in a directory is for file in *; do ……. Sep 18, 2013 at 23:05
  • @Gilles exactly; however i don't know why you added the comment 'ls' part to my answer, as i already mentioned that parsing ls will cause trouble (though i didn't bother to add a link to a more thorough discussion). i'll fix the while/for issue
    – umläute
    Sep 19, 2013 at 7:43
  • @umläute Are you suggest to compare the size of the folder instead of the size of the files in it?
    – ispasov
    Sep 19, 2013 at 7:48
  • no, i'm suggesting to compare the modification time of all files in the directory. (there's even an explanation why i consider comparing sizes a bad idea).
    – umläute
    Sep 19, 2013 at 7:50
  • 1
    Beware that stat -t \{\} | cut -d' ' -f13 will fail if the filename contains spaces. Use stat -c "%Y" \{\} instead (it's also easier to type and debug)
    – ndemou
    Apr 13, 2015 at 13:25
2

It's easier to control processing on the sending side where the process knows if the transfer is completed. This way you won't pick up aborted transfers either.

Send the file to a temporary name or directory. When the send is finished move the file into the correct location

cp srcvideo.avi /folder1/srcvideo.tmp && mv /folder1/srcvideo.tmp /folder1/srcvideo.avi

Then your script doesn't need to do all the testing on sizes. It can just wait for the correct file.

cd /folder1 || exit 1
for file in *.avi; do
  echo "$file found"
  do_some_processing "$file"
done

You could achieve the same by using a seperate transfer directory without changing filenames if you wish.

2
  • Well I am not the person who sends the files. Also they are sent from Windows PC. I need solution on my side of the process as a receiver.
    – ispasov
    Sep 19, 2013 at 6:40
  • You can still define a transfer spec for them. Another way is for them to transfer a 0 byte file at the end ${filename.done} that you poll for.
    – Matt
    Sep 19, 2013 at 8:36
1

Waiting for files to stop growing is a terrible way of detecting whether a download is finished. If the download pauses for too long due to a network glitch, your script will fire up.

The best way to do something upon the completion of a download is to either instruct the download program to run your script when the download is complete and successful, or to wait until the download program exits and then run your script if the download was successful. Any decent download program will allow at least one of these options.

If you are stuck with some inferior downloading method, use a notification facility to react when the file has finished downloading, instead of manually watching the file. On Linux, the notification facility is inotify. Depending on how the download program works, either react to a file closure (if the download program directly writes to the final file) or to a rename (if the download program first writes to a temporary file then renames it into place).

You can use the shell tool inotifywait to react when a file event occurs. Here's an example that converts every renamed file.

cd /path/to/directory
inotifywait -m -e moved_to --format=%f . |
while IFS= read -r filename; do
  conversion-program "$filename"
done
0

If you want to launch your script before you know if all the files are there, the first thing is to check if the number of file is constant. You could do something like, using listfile = $(ls "$folder")

[ $( echo $listfile | wc -w) != $(ls "$folder" | wc -w) ]

in a while loop; till the thing are not equal, your reset $listfile to $(ls "$folder"). Once it is done, you can simply do a doloop on $listfile where you do exactly what you were doing on one file; once it is done, you can launch the conversion.

1
  • This approach is prone to race conditions, especially when transferring large files. Also, the echo in echo $(ls "$folder") | wc -w is redundant. ls "$folder" | wc -w is sufficient. Sep 18, 2013 at 11:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.