4

I'm using find to prune old files, lots of them.. this takes minutes / hours to run and other server processes encounter IO performance issues. 

find -mtime +100 -delete -print

I tried ionice but it didn't appear to help. 

ionice -c 3

What can one do to 1. speed up the find operation and 2. to avoid impacting other processes? The FS is ext4.. is ext4 just bad at this kind of workload? Kernel is 3.16 Storage is 2x 1TB 7200rpm HDDs in RAID 1. There's 93GB in 610228 files now, so 152KB/file on average.

Maybe I just shouldn't store so many files in a single directory?

Improve this question
1
  • Add to the post, how many files, and disk technology, please.
    – Rui F Ribeiro
    Commented Nov 23, 2016 at 16:48

1 Answer 1

Reset to default
2

When you run the find command like you posted, it will do a rm for each file that it finds. This isn't a good way to do it, in terms of performance.

For improve this task, you can use the -exec option in find for process the output to a rm command:

find -mtime +100 -exec rm {} +

It's very important the use of the + termination instead the alternate \;. With +, find will only make a rm command for the maximum number of files it can process on a simple execution. With the \; termination, find will do a rm command for each file, so you would have the same problem.

For a better performance, you can join it to the ionice command like you mentioned. If you don't notice that it improves the system performance, most possible is that it is consuming other resources more than I/O, like CPU. For this, you can use renice command to decrease the priority in CPU usage of the process.

I would use the following:

ionice -c 3 find -mtime +100 -exec rm {} +

Now, in another shell, you need to find the PID of the find command: ps -ef | grep find

And finally run the renice command: renice +19 -p <PID_find_command>

Improve this answer
3
  • why not use xargs so that each spawn of a rm process deletes, say, 50 files at a time ?
    – steve
    Commented Nov 23, 2016 at 23:00
  • Using xargs will not work with files with blank spaces in the filename. You need to pay attention to build the comand with -print0 option that process the blank spaces, something like find -mtime +100 -print0 | xargs -0 rm. This complicates the execution of the command and will not work out of the box with the I/O performance modification in one single command, like in the example of ionice with find and -exec. Also, using xargs doesn't offer a better performance than using find with -exec and + termination, so I prefer this last option.
    – Rubén Alemán
    Commented Nov 24, 2016 at 23:01
  • Does -exec take care of spaces and other weird stuff? As we're IO bound, how does using rm improve performance?
    – XTF
    Commented Nov 29, 2016 at 17:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .