This question already has an answer here:

I have a directory where lots of cached files are getting generated very quickly. Since these are very small files, it is consuming all my inodes very quickly.

Currently I am running the following command to find all the files older than 12 hours and delete them.

$ find ./cache -mtime +0.5 -exec rm {} \;

But the rate at which this command is deleting is slower than the rate at which files are being generated. Can someone tell me some alternative way to remove large number of files quickly.

marked as duplicate by cas, garethTheRed, meuh, Archemar, Anthon May 18 '16 at 7:21

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • Maybe the problem is that it's not deleting enough files. You are aware that your command is deleting files whose mtime is more than 12 hours, right? If the files are generated very quickly, then you probably need to delete more recent files than that. – Joseph R. Oct 21 '13 at 9:20
  • Actually the command which @Gnouc mentioned worked faster. I dont know the reason how it works faster. – pradeepchhetri Oct 21 '13 at 9:43
  • 4
    Replace the \; with a + or better use -delete as you seem to be using GNU find. – Stéphane Chazelas Oct 21 '13 at 12:48

Try using xargs:

find ./cache -mtime +0.5 -print0 | xargs -0 rm -f

Update explaination for @pradeepchhetri

If you use find with -exec, every file that find found will call rm one time. So if you found a huge of files, i.e 10000 files, you called rm 10000 times.

xargs will treat ouput of find as command argument to rm, so that, xargs will provide as many arguments as rm can handle at once, i.e rm -f file1 file2 ... So it makes less fork call, make program run faster.

  • 1
    Can you explain me why this runs faster the command i mentioned. I noticed that it works faster. – pradeepchhetri Oct 21 '13 at 9:34
  • @pradeepchhetri With your method, find starts a new rm process for every file that meets the criteria. With Gnouc's method, xargs starts only one instance for rm for a bunch of files. Starting many less programs makes it faster. – kurtm Oct 21 '13 at 11:31
  • @kurtm: Thank you for the explanation. But i saw remarkable difference between the two commands, is the number of fork() syscalls is the only difference between the two ? – pradeepchhetri Oct 21 '13 at 12:06
  • @pradeepchhetri Yes, that is the only difference, the number of rm processes spawned. – kurtm Oct 21 '13 at 13:34
  • if you use find with \+ instead of \; it will bundle up the execs and will only launch an exec when the command line gets "big" – kdubs Feb 17 '17 at 18:31

find … -exec rm {} \; executes the rm command for each file. Even though starting a new process is pretty fast, it's still a lot slower than the mere act of deleting a file.

find … -exec rm {} + would call rm in batches, which is a lot faster: you pay the cost of running rm once per batch, and each batch performs many deletion.

Even faster is to not invoke rm at all. The find command on Linux has an action -delete to delete a matching file.

find ./cache -mtime +0.5 -delete

However, if you're producing files at such a rate that find … -exec rm {} \; can't keep up, there's probably something wrong with your setup. If cache contains millions of files, you should split it into subdirectories for faster access.

  • Nearly 7 million very small DHCP lease files consumed all the inodes. find -delete saved the day; removed about 6.5 million files in the same time as rm took to remove about 20 thousand. – studog Feb 27 '18 at 18:50

if the creation rate exceeds deletion rate you are best of by making the cache completely empty, and removing old files without any mtime evaluation

mv cache foobar
mkdir cache
# may require app restart
rm -rf foobar

Although find is the best (simplest, idiomatic) approach,

find $dir -exec rm {} +

You could move the directory aside, create a new directory (for your program), and then delete...

mv $idr old$dir && mkdir $dir && rm -rf old$dir

but maybe your problem is creating too many files. Why not change your program to append to an existing file, rather than create a new file? And then you could move this (logfile) aside, and then your program could create/append to a new file, for example,

fd = open("logfile","a+");

If you just want to get rid of many files as soon as possible ls -f1 /path/to/folder/with/many/files/ | xargs rm might work okay, but better don't run it on production systems because your system might become IO issues and applications might get stuck during the delete operation.

This script works nicely for many files and should not affect the ioload of the system.


# Path to folder with many files

# Temporary file to store file names

if [ -z "$FOLDER" ]; then
    echo "Prevented you from deleting everything! Correct your FOLDER variable!"
    exit 1

while true; do
    FILES=$(ls -f1 $FOLDER | wc -l)
    if [ "$FILES" -gt 10000 ]; then
        printf "[%s] %s files found. going on with removing\n" "$(date)" "$FILES"
        # Create new list of files
        ls -f1 $FOLDER | head -n 5002 | tail -n 5000 > "$FILE_FILENAMES"

        if [ -s $FILE_FILENAMES ]; then
            while read FILE; do
                rm "$FOLDER/$FILE"
                sleep 0.005
            done < "$FILE_FILENAMES"
        printf "[%s] script has finished, almost all files have been deleted" "$(date)"
    sleep 5

rm -rf directory/ also works faster for billion of files in one folder. I tried that.

  • But that would not prevent the program which generates the files to recreate the folder. Or it could make it fail, which is not a desirable effect. Apparently, the point here is to delete only old cached files, not all of them. – lgeorget Apr 17 '14 at 7:00

Another Linux specific solution would be to use inotify(7) facilities ; you'll detect when files are added, and then you'll immediately run something to remove the older ones.

OTOH, I guess that you might have some XY problem. Why do you have so many new files? Perhaps using sqlite, or GDBM indexed files, or some real database (e.g. PostGresQL, MariaDB, MongoDB) might be better.... Perhaps you need some version control system like git?

find . -name -mtime +0.5 -print -delete 

is another option to delete large number of files quickly.

Not the answer you're looking for? Browse other questions tagged or ask your own question.