Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a directory where lots of cached files are getting generated very quickly. Since these are very small files, it is consuming all my inodes very quickly.

Currently I am running the following command to find all the files older than 12 hours and delete them.

$ find ./cache -mtime +0.5 -exec rm {} \;

But the rate at which this command is deleting is slower than the rate at which files are being generated. Can someone tell me some alternative way to remove large number of files quickly.

share|improve this question
    
Maybe the problem is that it's not deleting enough files. You are aware that your command is deleting files whose mtime is more than 12 hours, right? If the files are generated very quickly, then you probably need to delete more recent files than that. –  Joseph R. Oct 21 '13 at 9:20
    
Actually the command which @Gnouc mentioned worked faster. I dont know the reason how it works faster. –  pradeepchhetri Oct 21 '13 at 9:43
1  
Replace the \; with a + or better use -delete as you seem to be using GNU find. –  Stéphane Chazelas Oct 21 '13 at 12:48

6 Answers 6

up vote 3 down vote accepted

Try using xargs:

find ./cache -mtime +0.5 -print0 | xargs -0 rm -f

Update explaination for @pradeepchhetri

If you use find with -exec, every file that find found will call rm one time. So if you found a huge of files, i.e 10000 files, you called rm 10000 times.

xargs will treat ouput of find as command argument to rm, so that, xargs will provide as many arguments as rm can handle at once, i.e rm -f file1 file2 ... So it makes less fork call, make program run faster.

share|improve this answer
1  
Can you explain me why this runs faster the command i mentioned. I noticed that it works faster. –  pradeepchhetri Oct 21 '13 at 9:34
    
@pradeepchhetri With your method, find starts a new rm process for every file that meets the criteria. With Gnouc's method, xargs starts only one instance for rm for a bunch of files. Starting many less programs makes it faster. –  kurtm Oct 21 '13 at 11:31
    
@kurtm: Thank you for the explanation. But i saw remarkable difference between the two commands, is the number of fork() syscalls is the only difference between the two ? –  pradeepchhetri Oct 21 '13 at 12:06
    
@pradeepchhetri Yes, that is the only difference, the number of rm processes spawned. –  kurtm Oct 21 '13 at 13:34

find … -exec rm {} \; executes the rm command for each file. Even though starting a new process is pretty fast, it's still a lot slower than the mere act of deleting a file.

find … -exec rm {} + would call rm in batches, which is a lot faster: you pay the cost of running rm once per batch, and each batch performs many deletion.

Even faster is to not invoke rm at all. The find command on Linux has an action -delete to delete a matching file.

find ./cache -mtime +0.5 -delete

However, if you're producing files at such a rate that find … -exec rm {} \; can't keep up, there's probably something wrong with your setup. If cache contains millions of files, you should split it into subdirectories for faster access.

share|improve this answer

Although find is the best (simplest, idiomatic) approach,

find $dir -exec rm {} +

You could move the directory aside, create a new directory (for your program), and then delete...

mv $idr old$dir && mkdir $dir && rm -rf old$dir

but maybe your problem is creating too many files. Why not change your program to append to an existing file, rather than create a new file? And then you could move this (logfile) aside, and then your program could create/append to a new file, for example,

fd = open("logfile","a+");
share|improve this answer

if the creation rate exceeds deletion rate you are best of by making the cache completely empty, and removing old files without any mtime evaluation

mv cache foobar
mkdir cache
# may require app restart
rm -rf foobar
share|improve this answer

rm -rf directory/ also works faster for billion of files in one folder. I tried that.

share|improve this answer
    
But that would not prevent the program which generates the files to recreate the folder. Or it could make it fail, which is not a desirable effect. Apparently, the point here is to delete only old cached files, not all of them. –  lgeorget Apr 17 at 7:00
find . -name -mtime +0.5 -print -delete 

is another option to delete large number of files quickly.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.