How can I immediately detect when new files added to a folder within a bash script? I would like the script to process files as soon as they are created in the folder. Are there any methods aside from scheduling a cron job that checks for new files each minute or so?
-
1Are you going to remove files from the folder once they are processed?– ztank1013Nov 19, 2011 at 20:00
-
possible duplicate of How to run a command when a directory's contents are updated?– Gilles 'SO- stop being evil'Nov 19, 2011 at 23:35
-
1actually the answers at that other question mentioned as possible duplicate are less diversified and not that helpful in case of cross-platform need. In fact for cross-platform I'd vote for fswatch (emcrisostomo.github.io/fswatch), also mentioned at a comment in another answer below– George BirbilisAug 15, 2020 at 11:48
12 Answers
You should consider using inotifywait
, as an example:
inotifywait -m /path -e create -e moved_to |
while read dir action file; do
echo "The file '$file' appeared in directory '$dir' via '$action'"
# do something with the file
done
In Ubuntu inotifywait
is provided by the inotify-tools
package. As of version 3.13 (current in Ubuntu 12.04) inotifywait
will include the filename without the -f option. Older versions may need to be coerced. What is important to note is that the -e
option to inotifywait
is the best way to do event filtering. Also, your read
command can assign the positional output into multiple variables that you can choose to use or ignore. There is no need to use grep/sed/awk to preprocess the output.
-
2
-
2Just want to update this. You do not need awk to achieve this. you can filter the events with '-e create' and get only the filename by doing '-f %f' or the full path using '-f %w%f'. So the first line of the above script becomes: inotifywait -m /path -f %w%f -e create |– LugouesMar 3, 2013 at 16:44
-
2@Lugoues and now when you try to use -f you get
The '--filename' option no longer exists. The option it enabled in earlier versions of inotifywait is now turned on by default.
So, you only have to doinotifywait -m /path -e create |
I'm going to try and edit this answer. Apr 17, 2014 at 6:27 -
2Now there is also a portable tool for it called
fswatch
. I did not write it, but it's open source and I use it.– user100278Jul 30, 2015 at 13:22 -
5@Wender inotfiywait outputs 3 pieces of information on a single line when triggered. The 'read' bash builtin reads the input line and assigns each of the three pieces of information to a variable. Thus the first piece is assigned to the variable path, the second to action, and the third to file. Having assigned values to those variables, they are then available to be used later (like on the echo line). More information: tldp.org/LDP/Bash-Beginners-Guide/html/sect_08_02.html– TimAug 4, 2018 at 13:38
In case you came here for a simple, fast, handy solution reading the title, you could use watch
watch -n 0.1 ls <your_folder>
Monitors your folder and lists you everything in it every 0.1 seconds
Drawback
Non-scriptable (For scripting options, have a look at other answers)
Not real time, so if a file was created and deleted in less than 0.1 second, then this would not work, watch
only supports minimum of 0.1 seconds.
-
1
-
I just cooked up this, and see no huge problems with it, other than a tiny chance of missing files in between checks.
while true
do
touch ./lastwatch
sleep 10
find /YOUR/WATCH/PATH -cnewer ./lastwatch -exec SOMECOMMAND {} \;
done
If your file processing doesn't take too long, you should not miss any new file. You could also background the activities... It's not bullet proof, but it serves some purposes without external tools like inotify.
-
Good catch. I improved it a bit to support spaces in filenames. Dec 16, 2015 at 7:34
-
Absolutely. That's the way to go. Not really sure why I went down that road, I use -exec routinely. Dec 26, 2015 at 18:44
-
-
5Best solution if
inotify
is not available. I would add-type f
to filter out files only. Otherwise the folder will also be returned. May 31, 2017 at 4:46 -
Yep - the
-f filename
option is great. So then the only remaining question is how to get this to start upon reboot. I am going to use this with my solar plant toos.system("ssh me@mysystem ' ( touch /home/me/alarms/low24 ) '")
so then the creation of this file will cause the master computer to useespeak
and announce the low voltage. It already sends me an email but since my system already speaks the time at the top of the hour it has all the rest. askubuntu.com/questions/977613/…– SDsolarNov 20, 2017 at 0:21
I prefer incron
, as its easier to manage. Essentially it's a service that leverages inotify
and you can setup configurations to take action based on file change operations.
Ex:
<directory> <file change mask> <command or action> options
/var/www/html IN_CREATE /root/scripts/backup.sh
You can see a full example here: http://www.cyberciti.biz/faq/linux-inotify-examples-to-replicate-directories/
-
incron never worked for me. based on SE questions about it, it seems like many people have unresolved issues with it.– bobNov 8, 2019 at 14:38
entr
Using entr
is the new way to do this (it's cross platform). Note entr
doesn't use polling giving it a huge advantage over many of the alternatives.
Uses
kqueue(2)
orinotify(7)
to avoid polling.entr
was written to make rapid feedback and automated testing natural and completely ordinary.
On BSD it uses pledge(2)
You can install it with
apt-get install entr
dnf install entr
brew install entr
You can track a directory for new additions using
while $(true); do
# echo ./my_watch_dir | entr -dnr echo "Running trigger..."
echo ./my_watch_dir | entr -dnr ##MY COMMAND##
done;
Options explained (from the docs),
-d
Track the directories of regular files provided as input and exit if a new file is added. This option also enables directories to be specified explicitly. Files with names beginning with ‘.’ are ignored.-n
Run in non-interactive mode. In this mode entr does not attempt to read from the TTY or change its properties.-r
Reload a persistent child process. As with the standard mode of operation, a utility which terminates is not executed again until a file system or keyboard event is processed.SIGTERM
is used to terminate the utility before it is restarted. A process group is created to prevent shell scripts from masking signals.entr
waits for the utility to exit to ensure that resources such as sockets have been closed. Control of the TTY is not transferred the child process.
Note entr
isn't polling there. It's after there is an operation on an inode it hasn't seen. Note this isn't a failsafe mechanism because the inode could be recycled to a different file name entirely. Ie,
mkdir foo;
cd foo;
touch bar;
touch baz;
ls -1i;
echo "delete bar; add quz"
rm bar;
touch quz;
ls -1i;
See the inotifywait
answer for a better, and more powerful method of doing this. Albeit with a much worse interface.
-
maybe this is a dumb question, but how can it achieve that without using polling– Thanos MJul 22, 2021 at 19:42
-
1
-
entr
is great to restart a process when files changes, but it's not able to give you the name of the file that changed or was added to a folder: it gives the first file in the folder that was modified, which is not useful– MrEFeb 11 at 1:59
I am assuming the target folder (I'll call it isempty
just for convenience) is empty and you are waiting for one or more files to be dropped there.
You can use the following command:
ls -1A isempty | wc -l
just to check if the folder is still empty, in fact it will return a 0 if there is no new file (hence the isempty
folder is still empty) or, on the other hand, it will return a value greater than 0 (actually the number of files currently in the folder).
That said a silly if/then test can make the rest of the work:
if [ $(ls -1A isempty | wc -l) -gt 0 ] ; then do_something ; fi
Of course the do_something
function will have to manipulate the file(s) within the isempty
folder and then remove it(them) from the folder itself after processing.
Adding a line like the following in your crontab will run the check once a minute and will trigger the do_something
action if the folder is not empty of course:
* * * * * if [ $(ls -1A isempty | wc -l) -gt 0 ] ; then do_something ; fi
-
This solution works for mounted remote filesystems. inotify-tools developer(s) is working on fuse (or was in mid 2014).– RondoMay 26, 2015 at 0:07
-
3You shouldn't ever use
ls
for scripting. Usefind
or simple globbing instead: mywiki.wooledge.org/ParsingLs– andsensNov 28, 2016 at 23:05
If you want to detect new files, then process them and at the end delete proceeded files you can use systemd.path. This method bases on inotify. There is an option DirectoryNotEmpty, so systemd can run your script always when it detects any files in directory. You have to remember it will work only if you can delete proceeded files and script leaves directory empty.
First prepare mymonitor.service file
[Unit]
Description=Start the script
[Service]
Type=oneshot
ExecStart=/path/to/your/script
next go to mymonitor.path to define the path
[Unit]
Description= Triggers the service
[Path]
DirectoryNotEmpty=/path/to/monitor
[Install]
WantedBy=multi-user.target
If the name of the .path file is the same as the name of the service there is no need to specify the service name in .path file.
It bases on Monitoring File Access for Dummies
-
1
-
After searching the answers here and the ones at the other question mentioned as possible duplicate in the comments above, I think I'd go with fswatch since it is has cross-platform support.
It uses different kinds of monitors for different OS and can choose the appropriate one automatically, or allow one to specify which one to use and even pass custom platform-specific parameters to the respective monitor.
The list of monitors it currently supports is:
- A monitor based on the File System Events API of Apple OS X.
- A monitor based on kqueue, an event notification interface introduced in FreeBSD 4.1 and supported on most *BSD systems (including OS X).
- A monitor based on inotify, a Linux kernel subsystem that reports file system changes to applications.
- A monitor based on File Events Notification, a Solaris/Illumos kernel API that reports file events.
- A monitor based on ReadDirectoryChangesW, a Microsoft Windows API that reports changes to a directory.
- A monitor which periodically stats the file system, saves file modification times in memory and manually calculates file system changes, which can work on any operating system where stat (2) can be used.
It seems to be available via apt-get on Debian/Ubuntu Linuxes. See how to install via apt-get and use fswatch.
FreeBSD and OS-X package-based installation support for fswatch is provided by its author.
Can also build and install it at other OSes, found an article+video showing how to make and install fswatch on CentOS.
There is also an other article that shows the same manual process to build/install (and use) fswatch for Linux.
Windows-based package-based installation support doesn't seem to be available yet (e.g. no package for Chocolatey yet, and no package for Vcpkg)
There is also extensive documentation for fswatch, though docs for the latest 1.5 version point to 1.4 ones currently. See how to convert commands for fswatch 0.x to fswatch 1.x
Tips for manually choosing a monitor (currently not updated to mention all monitors)
Read about fswatch usage here, here and here and a tutorial intro here
A library named libfswatch is kept in sync with the fswatch tool. See here and a newer doc here. Note that the library is versioned differently from fswatch utility itself. Specifically, the 1.14.0 library doc states:
"libtool's versioning scheme is described by three integers: current:revision:age.
- current is the most recent interface number implemented by the library.
- revision is the implementation number of the current interface.
- age is the difference between the newest and the oldest interface that the library implements.
Beware that there is also another similar s/w called fswatch (that is go-related I think).
-
For Solaris do note the comment "The behaviour depends on the monitor. Solaris File Events Notification will tell you that a directory has changed when a file has been created, but won't give you any more details. It's up to the receiver to scan the directory contents and see what file has been created." at github.com/emcrisostomo/fswatch/issues/228 Aug 15, 2020 at 18:20
-
Also, note the license is GPL v3.0, so you should avoid statically linking to libfswatch (but dynamically link to its shared library instead) or prefer using the fswatch utility instead if you don't want to be forced to share the source code of your app too. Regarding having to share fswatch's source code, guess sharing a URL to its source code repository should be enough (that one has the most up to date version of the code anyway). Aug 15, 2020 at 18:23
Bash cannot do this easily. You'd have to basically get a list of all the files in the folder and periodically get a new list and compare them to see whats changed.
What you're looking for is called inotify. Its built into the linux kernel and you can basically sit there waiting for something to happen at which point inotify comes back and says 'hey, theres a new file called foobar'
To accomplish what you want you'd have to switch to something like perl and use Linux::Inotify2 (python probably supports inotify as well, but I'm a perl person).
This works in cygwin and Linux. Some of the previous solutions which write a file will cause the disk to thrash. This scipt does not have that problem:
SIG=1
SIG0=$SIG
while [ $SIG != 0 ] ; do
while [ $SIG = $SIG0 ] ; do
SIG=`ls -1 | md5sum | cut -c1-32`
sleep 10
done
SIG0=$SIG
ls -lrt | tail -n 1
done
Below is an abridged version of example on stackoverflow that I've tested and incorporated into one of my projects that requires monitoring of specific directories.
Var_dir="${1:-/tmp}"
Var_diff_sleep="${2:-120}"
Var_diff_opts="--suppress-common-lines"
Func_parse_diff(){
_added="$(grep -E '>' <<<"${@}")"
if [ "${#_added}" != "0" ]; then
mapfile -t _added_list <<<"${_added//> /}"
_let _index=0
until [ "${#_added_list[@]}" = "${_index}" ]; do
_path_to_check="${Var_dir}/${_added_list[${_index}]}"
if [ -f "${_path_to_check}" ]; then
echo "# File: ${_path_to_check}"
elif [ -d "${_path_to_check}" ]; then
echo "# Directory: ${_path_to_check}"
if [ -p "${_path_to_check}" ]; then
echo "# Pipe: ${_path_to_check}"
fi
let _index++
done
unset _index
fi
}
Func_watch_bulk_dir(){
_current_listing=""
while [ -d "${Var_dir}" ]; do
_new_listing="$(ls "${Var_dir}")"
_diff_listing="$(diff ${Var_dec_diff_opts} <(${Var_echo} "${_current_listing}") <(${Var_echo} "${_new_listing}"))"
if [ "${_diff_listing}" != "0" ]; then
Func_parse_diff "${_diff_listing}"
fi
_current_listing="${_new_listing}"
sleep ${Var_diff_sleep}
done
}
Here's a link to a script that uses a modified version of above to automatically decrypt files or directories found in its sshfs mount point; the afore mentioned project.
See also Watchman (based on inotify), especially when complex monitoring and action-triggering is involved. It's available for not just Linux, but also Windows and Mac. See more at A: Watchman Q: How to monitor a complete directory tree for changes in Linux?.