Tell me more ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.
up vote 7 down vote favorite
3

We have an issue with a folder becoming unwieldy with hundreds of thousands of tiny files.

There are so many files that performing rm -rf returns an error and instead what we need to do is something like:

find /path/to/folder -name "filenamestart*" -type f -exec rm -f{} \;

This works but is very slow and constantly fails from running out of memory.

Is there a better way to do this? Ideally I would like to remove the entire directory without caring about the contents inside it.

share|improve this question
5  
rm -rf * in the folder probably fails because of too many arguments; but what about rm -rf folder/ if you want to remove the entire directory anyways? – sr_ Apr 26 '12 at 8:01
From memory that is what I was doing, I think because it recurses in to build out the list of files to delete before it deletes them? – Toby Apr 26 '12 at 8:09
Just out of curiosity - how many files does it take to break rm -rf? – jw013 Apr 26 '12 at 11:37
2  
You should probably rename the question to something more accurate, like "Efficiently delete large directory containing thousands of files." In order to delete a directory and its contents, recursion is necessary by definition. You could manually unlink just the directory inode itself (probably requires root privileges), unmount the file system, and run fsck on it to reclaim the unused disk blocks, but that approach seems risky and may not be any faster. In addition, the file system check might involve recursively traversing the file system tree anyways. – jw013 Apr 26 '12 at 13:27
Once I had a ccache file tree so huge, and rm was taking so long (and making the entire system sluggish), it was considerably faster to copy all other files off the filesystem, format, and copy them back. Ever since then I give such massive small file trees their own dedicated filesystem, so you can mkfs directly instead of rm. – frostschutz Jun 15 at 11:43

8 Answers

up vote 9 down vote

What about something like: find /path/to/folder -name "filenamestart*" -type f -print0 | xargs -0rn 20 rm -f

You can limit number of files to delete at once by changing the argument for parameter -n. The file names with blanks are included also.

share|improve this answer
You probably don't need the -n 20 bit, since xargs should limit itself to acceptable argument-list sizes anyway. – Useless Apr 26 '12 at 13:41
Yes, you are right. Here is a note from man xargs : (...) max-chars characters per command line (...). The largest allowed value is system-dependent, and is calculated as the argument length limit for exec. So -n option is for such cases where xargs cannot determine the CLI buffer size or if the executed command has some limits. – digital_infinity Apr 26 '12 at 13:50
up vote 7 down vote

Someone on Twitter suggested using -delete instead of -exec rm -f{} \;

This has improved the efficiency of the command, it still uses recursion to go through everything though.

share|improve this answer
6  
This is non standard. GNU find have -delete, and other find maybe. – enzotib Apr 26 '12 at 9:11
Thanks for the comment @enzotib I didn't know that! – Toby Apr 26 '12 at 9:21
5  
-delete should always be preferred to -exec rm when available, for reasons of safety and efficiency. – jw013 Apr 26 '12 at 11:37
up vote 4 down vote

Instead of deleting it manually, I suggest having the folder on a separate partition and simply unmount && format && remount.

share|improve this answer
This makes sense going forward, thanks for the answer. – Toby Apr 26 '12 at 14:56
up vote 2 down vote

Expanding on one of the comments, I do not think you're doing what you think you're doing.

First I created a huge amount of files, to simulate your situation:

$ mkdir foo
$ cd foo/
$ for X in $(seq 1 1000);do touch {1..1000}_$X; done

Then I tried what I expected to fail, and what it sounds like you're doing in the question:

$ rm -r foo/*
bash: /bin/rm: Argument list too long

But this does work:

$ rm -r foo/
$ ls foo
ls: cannot access foo: No such file or directory
share|improve this answer
up vote 0 down vote 
ls -1 | xargs rm -rf

should work inside the main folder

share|improve this answer
ls won't work because of the amount of files in the folder. This is why I had to use find, thanks though. – Toby Apr 26 '12 at 8:19
4  
@Toby: Try ls -f, which disables sorting. Sorting requires that the entire directory be loaded into memory to be sorted. An unsorted ls should be able to stream its output. – camh Apr 26 '12 at 10:59
up vote 0 down vote

Deleting REALLY LARGE directories needs a different approach, as I learned from this site - you'll need to utilize ionice.It ensures (with -c3) that deletes will only be performed when the system has IO-time for it. You systems load will not rise to high and everything stays responsive (though my CPU time for find was quite high at about 50%). 

find <dir> -type f -exec ionice -c3 rm {} \;
share|improve this answer
up vote 0 down vote

There are couple of methods that can be used to delete large number of files in linux,. You can use find with delete option, which is faster than exec option. Then you can use perl unlink, then even rsync. How to delete large number of files in linux

share|improve this answer
up vote 0 down vote

Using rsync is surprising fast and simple.

mkdir empty_dir
rsync -a --delete empty_dir/    yourdirectory/

Source: http://stackoverflow.com/questions/1795370/unix-fast-remove-directory-for-cleaning-up-daily-builds

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.