Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

We have an issue with a folder becoming unwieldy with hundreds of thousands of tiny files.

There are so many files that performing rm -rf returns an error and instead what we need to do is something like:

find /path/to/folder -name "filenamestart*" -type f -exec rm -f{} \;

This works but is very slow and constantly fails from running out of memory.

Is there a better way to do this? Ideally I would like to remove the entire directory without caring about the contents inside it.

share|improve this question
7  
rm -rf * in the folder probably fails because of too many arguments; but what about rm -rf folder/ if you want to remove the entire directory anyways? –  sr_ Apr 26 '12 at 8:01
    
From memory that is what I was doing, I think because it recurses in to build out the list of files to delete before it deletes them? –  Toby Apr 26 '12 at 8:09
    
Instead of deleting it manually, I suggest having the folder on a separate partition and simply unmount && format && remount. –  bbaja42 Apr 26 '12 at 11:22
4  
You should probably rename the question to something more accurate, like "Efficiently delete large directory containing thousands of files." In order to delete a directory and its contents, recursion is necessary by definition. You could manually unlink just the directory inode itself (probably requires root privileges), unmount the file system, and run fsck on it to reclaim the unused disk blocks, but that approach seems risky and may not be any faster. In addition, the file system check might involve recursively traversing the file system tree anyways. –  jw013 Apr 26 '12 at 13:27
2  
Once I had a ccache file tree so huge, and rm was taking so long (and making the entire system sluggish), it was considerably faster to copy all other files off the filesystem, format, and copy them back. Ever since then I give such massive small file trees their own dedicated filesystem, so you can mkfs directly instead of rm. –  frostschutz Jun 15 '13 at 11:43

12 Answers 12

Someone on Twitter suggested using -delete instead of -exec rm -f{} \;

This has improved the efficiency of the command, it still uses recursion to go through everything though.

share|improve this answer
7  
This is non standard. GNU find have -delete, and other find maybe. –  enzotib Apr 26 '12 at 9:11
    
Thanks for the comment @enzotib I didn't know that! –  Toby Apr 26 '12 at 9:21
10  
-delete should always be preferred to -exec rm when available, for reasons of safety and efficiency. –  jw013 Apr 26 '12 at 11:37

What about something like: find /path/to/folder -name "filenamestart*" -type f -print0 | xargs -0rn 20 rm -f

You can limit number of files to delete at once by changing the argument for parameter -n. The file names with blanks are included also.

share|improve this answer
    
You probably don't need the -n 20 bit, since xargs should limit itself to acceptable argument-list sizes anyway. –  Useless Apr 26 '12 at 13:41
    
Yes, you are right. Here is a note from man xargs : (...) max-chars characters per command line (...). The largest allowed value is system-dependent, and is calculated as the argument length limit for exec. So -n option is for such cases where xargs cannot determine the CLI buffer size or if the executed command has some limits. –  digital_infinity Apr 26 '12 at 13:50

Using rsync is surprising fast and simple.

mkdir empty_dir
rsync -a --delete empty_dir/    yourdirectory/

Source: http://stackoverflow.com/questions/1795370/unix-fast-remove-directory-for-cleaning-up-daily-builds

share|improve this answer

Expanding on one of the comments, I do not think you're doing what you think you're doing.

First I created a huge amount of files, to simulate your situation:

$ mkdir foo
$ cd foo/
$ for X in $(seq 1 1000);do touch {1..1000}_$X; done

Then I tried what I expected to fail, and what it sounds like you're doing in the question:

$ rm -r foo/*
bash: /bin/rm: Argument list too long

But this does work:

$ rm -r foo/
$ ls foo
ls: cannot access foo: No such file or directory
share|improve this answer
2  
This is the only solution that worked: Run rm -Rf bigdirectory several times. I had a directory with thousands of millions of subdirectories and files. I couldn’t even run ls or find or rsync in that directory, because it ran out of memory. The command rm -Rf quit many times (out of memory) only deleting part of the billions of files. But after many retries it finally did the job. Seems to be the only solution if running out of memory is the problem. –  erik Apr 9 at 13:01

I had the opportunity to test -delete as compared to -exec rm \{\} \; and for me -delete was the answer to this problem.

Using -delete deleted the files in a folder of 400,000 files at least 1,000 times faster than rm.

The 'How to delete large number of files in linux' article suggests it is about three time faster, but in my test the difference was much more dramatic.

share|improve this answer

I tried to comment on the -delete option above, but my reputation isn't high enough, so I'll simply post that I'm using it to remove a large number (1M+ est) files in a temp folder that I created and inadvertently forgot to cleanup nightly. I filled my disk/partition accidentally, and nothing else could remove them but the find . command. It is slow, at first I was using:

find . -ls -exec rm {} \;

But that was taking an EXTREME amount of time. It started after about 15 mins to remove some of the files, but my guess is that it was removing less than 10 or so per second after it finally started. So, I tried the:

find . -delete

instead, and I'm letting it run right now. It appears to be running faster, though it's EXTREMELY taxing on the CPU which the other command was not. It's been running for like an hour now and I think I'm getting space back on my drive and the partition gradually "slimming down" but it's still taking a very long time. I seriously doubt it's running 1,000 times faster than the other. As in all things, I just wanted to point out the tradeoff in space vs. time. If you have the CPU bandwidth to spare (we do) then run the latter. It's got my CPU running (uptime reports):

10:59:17 up 539 days, 21:21,  3 users,  load average: 22.98, 24.10, 22.87

And I've seen the load average go over 30.00 which is not good for a busy system, but for ours which is normally lightly loaded, it's OK for a couple hours. I've checked most other things on the system and they're still responsive so we are OK for now.

share|improve this answer
    
if you're going to use exec you almost certainly want not use -ls and do find . -type f -exec rm '{}' + + is faster because it will give as many arguments to rm as it can handle at once. –  xenoterracide Jan 3 at 17:48
    
I think you should go ahead and edit this into its own answer… it's really too long for a comment. Also, it sound like your filesystem has fairly expensive deletes, curious which one it is? You can run that find … -delete through nice or ionice, that may help. So might changing some mount options to less-crash-safe settings. (And, of course, depending on what else is on the filesystem, the quickest way to delete everything is often mkfs.) –  derobert Jan 4 at 7:24
    
Load average is not always CPU, it's just a measure of the number of blocked processes over time. Processes can block on disk I/O, which is likely what is happening here. –  Score_Under Jul 14 at 12:47
ls -1 | xargs rm -rf 

should work inside the main folder

share|improve this answer
    
ls won't work because of the amount of files in the folder. This is why I had to use find, thanks though. –  Toby Apr 26 '12 at 8:19
4  
@Toby: Try ls -f, which disables sorting. Sorting requires that the entire directory be loaded into memory to be sorted. An unsorted ls should be able to stream its output. –  camh Apr 26 '12 at 10:59
    
Does not work on filenames that contain newlines. –  maxschlepzig Jan 5 at 7:53

Deleting REALLY LARGE directories needs a different approach, as I learned from this site - you'll need to utilize ionice.It ensures (with -c3) that deletes will only be performed when the system has IO-time for it. You systems load will not rise to high and everything stays responsive (though my CPU time for find was quite high at about 50%).

find <dir> -type f -exec ionice -c3 rm {} \;
share|improve this answer
1  
using + instead of \; would make this faster as it passes more arguments to rm at once, less forking –  xenoterracide Jan 3 at 17:50

There are couple of methods that can be used to delete large number of files in linux,. You can use find with delete option, which is faster than exec option. Then you can use perl unlink, then even rsync. How to delete large number of files in linux

share|improve this answer

For Izkata's hint above:

But this does work:

$ rm -r foo/
$ ls foo
ls: cannot access foo: No such file or directory

This almost worked - or would have worked - but I had some problems in permission; files were on a server, but still I don't understand where this permission issue came from. Anyway, Terminal asked for confirmation on every file. Amount of files was around 20 000, so this wasn't an option. After "-r" I added option "-f", so the whole command was "rm -r -f foldername/". Then it seemed to work fine. I'm a novice with Terminal, but I guess this was okay, right? Thanks!

share|improve this answer

Depending on how well you need to get rid of those files, I'd suggest using shred.

$ shred -zuv folder

if you want to purge the directory, but you can't remove it and recreate it, I suggest moving it and recreating it instantly.

mv folder folder_del
mkdir folder
rm -rf folder_del

this is faster, believe it or not, as only one inode has to be changed. Remember: You can't really parallelize this tast on a multicore computer. It comes down to disk access, which is limited by the RAID or what have you.

share|improve this answer
    
shred will not work with many modern filesystems. –  paraxor Jul 2 '13 at 14:47

A clever trick:

rsync -a --delete empty/ your_folder/

It's super CPU intensive, but really really fast. See http://linuxnote.net/jianingy/en/linux/a-fast-way-to-remove-huge-number-of-files.html

share|improve this answer

protected by slm Feb 18 at 21:21

Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.