0

Trying to use split with inotifywait which basically splits the file when the file is created from ftp-server.

#!/bin/bash

TARGET=/home/test-directory/incoming
SPLITTED=/home/test-directory/incoming/splitted
PROCESSED=/home/test-directory/incoming/processed
LOGFILE=/var/log/inotify-ftp.log

inotifywait -m -e create -e moved_to --format "%f" $TARGET \
        | while read FILENAME
                do
                        echo Detected $FILENAME >> $LOGFILE
                        echo Splitting $FILENAME >> $LOGFILE
                        split -d -l 1000 "$TARGET/$FILENAME" "$SPLITTED/$FILENAME"
                        #/usr/bin/split -d -l 1000 /home/test-directory/incoming/test-file.csv /home/test-directory/incoming/splitted/test-file.csv
                        mv "$TARGET/$FILENAME" "$PROCESSED/$FILENAME"
                        echo Completed splitting $FILENAME >> $LOGFILE
                done

So, the following code works fine when its executed separated. The above script is supposed to do the same thing but instead, it creates a first splitted file with couple hundred lines only.

#/usr/bin/split -d -l 1000 /home/test-directory/incoming/test-file.csv /home/test-directory/incoming/splitted/test-file.csv

Any idea, whats going on ?

2

That would be because the file is zero-length when it is created. There is a race condition, where split figures out the size of the file, and decides how to split it up, while ftp-server is happily making it bigger all the time.

It would be wise to figure a mechanism where the split waits for the file to arrive completely before starting to read it. Typically, stat the file in a loop until it has not grown in the previous minute.

|improve this answer|||||
  • 2
    Use close_write instead of create – roaima Dec 20 '19 at 0:07
1

As has been explained elsewhere the create event for inotifywait fires when the file has (just) been created, with no reference to content being added. Change the create event to close_write so that you don't get a trigger until the file has been closed by your file transfer service:

inotifywait -m -e close_write -e moved_to --format "%f" "$TARGET"

You should also build in some error handling so that your process can differentiate between a completely transferred file and one that's been interrupted part way through. This might be checking the validity of the file contents, or by having the file transferred with a tmpsuffix that's removed by the sender renaming the file on successful transfer, etc.

Finally, as good coding practice remember to double-quote your variables everywhere you use them.

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.