1

I want to recursively delete all files that end with .in. This is taking a long time, and I have many cores available, so I would like to parallelize this process. From this thread, it looks like it's possible to use xargs or make to parallelize find. Is this application of find possible to parallelize?

Here is my current serial command:

find . -name "*.in" -type f -delete
  • This is very likely I/O bound, and not CPU bound, so adding more cores probably won't help. If your find spans several harddisks, just do a find for each one. – dirkt Mar 7 '17 at 8:05
3

Replacing -delete with -print (which is the default) and piping into GNU parallel should mostly do it:

find . -name '*.in' -type f | parallel rm --

This will run one job per core; use -j N to use N parallel jobs instead.

It's not completely obvious that this will run much faster than deleting in sequence, since deleting is probably more I/O- than CPU-bound, but it would be interesting to test out.

(I said "mostly do it" because the two commands are not fully equivalent; for example, the parallel version will not do the right thing if some of your input paths include newline characters.)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.