How to detect new files in a folder with a script? I would like to process the files as soon as they are created in the folder. Is this possible to do so or do I have to schedule a script with that checks for new files each minute or so?

up vote 124 down vote accepted

You should consider using inotifywait, as an example:

inotifywait -m /path -e create -e moved_to |
    while read path action file; do
        echo "The file '$file' appeared in directory '$path' via '$action'"
        # do something with the file
    done

In Ubuntu inotifywait is provided by the inotify-tools package. As of version 3.13 (current in Ubuntu 12.04) inotifywait will include the filename without the -f option. Older versions may need to be coerced. What is important to note is that the -e option to inotifywait is the best way to do event filtering. Also, your read command can assign the positional output into multiple variables that you can choose to use or ignore. There is no need to use grep/sed/awk to preprocess the output.

  • 1
    Great! The inotifywait was just what I wanted. – ihatetoregister Nov 19 '11 at 18:50
  • Why is fflush() necessary here? Since a newline is already created I think – daisy Oct 18 '12 at 2:52
  • 2
    Just want to update this. You do not need awk to achieve this. you can filter the events with '-e create' and get only the filename by doing '-f %f' or the full path using '-f %w%f'. So the first line of the above script becomes: inotifywait -m /path -f %w%f -e create | – Lugoues Mar 3 '13 at 16:44
  • 2
    @Lugoues and now when you try to use -f you get The '--filename' option no longer exists. The option it enabled in earlier versions of inotifywait is now turned on by default. So, you only have to do inotifywait -m /path -e create | I'm going to try and edit this answer. – Bruno Bronosky Apr 17 '14 at 6:27
  • 1
    Now there is also a portable tool for it called fswatch. I did not write it, but it's open source and I use it. – user100278 Jul 30 '15 at 13:22

I prefer incron, as its easier to manage. Essentially it's a service that leverages inotify and you can setup configurations to take action based on file change operations.

Ex:

<directory> <file change mask> <command or action>  options
/var/www/html IN_CREATE /root/scripts/backup.sh

You can see a full example here: http://www.cyberciti.biz/faq/linux-inotify-examples-to-replicate-directories/

I just cooked up this, and see no huge problems with it, other than a tiny chance of missing files in between checks.

while true
do
       touch  ./lastwatch
       sleep 10
       find /YOUR/WATCH/PATH -cnewer ./lastwatch -exec SOMECOMMAND {} \;
done

If your file processing doesn't take too long, you should not miss any new file. You could also background the activities... It's not bullet proof, but it serves some purposes without external tools like inotify.

  • Good catch. I improved it a bit to support spaces in filenames. – Michael Sacchi Dec 16 '15 at 7:34
  • Absolutely. That's the way to go. Not really sure why I went down that road, I use -exec routinely. – Michael Sacchi Dec 26 '15 at 18:44
  • its not realtime. realtime is always best – Farhan Jun 13 '16 at 1:29
  • 2
    Best solution if inotify is not available. I would add -type f to filter out files only. Otherwise the folder will also be returned. – Xiao Peng - ZenUML.com May 31 '17 at 4:46
  • Yep - the -f filename option is great. So then the only remaining question is how to get this to start upon reboot. I am going to use this with my solar plant to os.system("ssh me@mysystem ' ( touch /home/me/alarms/low24 ) '") so then the creation of this file will cause the master computer to use espeak and announce the low voltage. It already sends me an email but since my system already speaks the time at the top of the hour it has all the rest. askubuntu.com/questions/977613/… – SDsolar Nov 20 '17 at 0:21

I am assuming the target folder (I'll call it isempty just for convenience) is empty and you are waiting for one or more files to be dropped there.

You can use the following command:

ls -1A isempty | wc -l

just to check if the folder is still empty, in fact it will return a 0 if there is no new file (hence the isempty folder is still empty) or, on the other hand, it will return a value greater than 0 (actually the number of files currently in the folder).

That said a silly if/then test can make the rest of the work:

if [ $(ls -1A isempty | wc -l) -gt 0 ] ; then do_something ; fi

Of course the do_something function will have to manipulate the file(s) within the isempty folder and then remove it(them) from the folder itself after processing.

Adding a line like the following in your crontab will run the check once a minute and will trigger the do_something action if the folder is not empty of course:

* * * * *     if [ $(ls -1A isempty | wc -l) -gt 0 ] ; then do_something ; fi
  • This solution works for mounted remote filesystems. inotify-tools developer(s) is working on fuse (or was in mid 2014). – Rondo May 26 '15 at 0:07
  • 2
    You shouldn't ever use ls for scripting. Use find or simple globbing instead: mywiki.wooledge.org/ParsingLs – andsens Nov 28 '16 at 23:05

If you want to detect new files, then process them and at the end delete proceeded files you can use systemd.path. This method bases on inotify. There is an option DirectoryNotEmpty, so systemd can run your script always when it detects any files in directory. You have to remember it will work only if you can delete proceeded files and script leaves directory empty.

First prepare mymonitor.service file

[Unit]
Description=Start the script

[Service]
Type=oneshot
ExecStart=/path/to/your/script

next go to mymonitor.path to define the path

[Unit]
Description= Triggers the service

[Path]
DirectoryNotEmpty=/path/to/monitor

[Install]
WantedBy=multi-user.target

If the name of the .path file is the same as the name of the service there is no need to specify the service name in .path file.

It bases on Monitoring File Access for Dummies

You can use watch in your script

watch -n 0.1 ls <your_folder>

Monitors your folder and lists you everything in it every 0.1 seconds

Drawback

Is not real time, so if a file was created and deleted in less than 0.1 second, then this would not work, watch only supports minimum of 0.1 seconds.

Bash cannot do this easily. You'd have to basically get a list of all the files in the folder and periodically get a new list and compare them to see whats changed.

What you're looking for is called inotify. Its built into the linux kernel and you can basically sit there waiting for something to happen at which point inotify comes back and says 'hey, theres a new file called foobar'

To accomplish what you want you'd have to switch to something like perl and use Linux::Inotify2 (python probably supports inotify as well, but I'm a perl person).

This works in cygwin and Linux. Some of the previous solutions which write a file will cause the disk to thrash. This scipt does not have that problem:

SIG=1
SIG0=$SIG
while [ $SIG != 0 ] ; do
 while [ $SIG = $SIG0 ] ; do
   SIG=`ls -1 | md5sum | cut -c1-32`
   sleep 10
 done
 SIG0=$SIG
 ls -lrt | tail -n 1
done

Below is an abridged version of example on stackoverflow that I've tested and incorporated into one of my projects that requires monitoring of specific directories.

Var_dir="${1:-/tmp}"
Var_diff_sleep="${2:-120}"
Var_diff_opts="--suppress-common-lines"
Func_parse_diff(){
    _added="$(grep -E '>' <<<"${@}")"
    if [ "${#_added}" != "0" ]; then
        mapfile -t _added_list <<<"${_added//> /}"
        _let _index=0
        until [ "${#_added_list[@]}" = "${_index}" ]; do
            _path_to_check="${Var_dir}/${_added_list[${_index}]}"
            if [ -f "${_path_to_check}" ]; then
                echo "# File: ${_path_to_check}"
            elif [ -d "${_path_to_check}" ]; then
                echo "# Directory: ${_path_to_check}"
            if [ -p "${_path_to_check}" ]; then
                echo "# Pipe: ${_path_to_check}"
            fi
            let _index++
        done
        unset _index
    fi
}
Func_watch_bulk_dir(){
    _current_listing=""
    while [ -d "${Var_dir}" ]; do
        _new_listing="$(ls "${Var_dir}")"
        _diff_listing="$(diff ${Var_dec_diff_opts} <(${Var_echo} "${_current_listing}") <(${Var_echo} "${_new_listing}"))"
        if [ "${_diff_listing}" != "0" ]; then
            Func_parse_diff "${_diff_listing}"
        fi
        _current_listing="${_new_listing}"
        sleep ${Var_diff_sleep}
    done
}

Here's a link to a script that uses a modified version of above to automatically decrypt files or directories found in its sshfs mount point; the afore mentioned project.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.