7

Is there some mechanism to make find with -exec use multiple jobs?

Lets use this command for example

find ./* -exec flac --best {} \;

Is there some way to set the number of concurrent -exec commands to run easier than something like this solution:

run find ./* and save the results to a file/pipe them for another command to split into 16 parts (or however many threads you want) then run commands on these?

4
  • 2
    Pipe to xargs and use -P
    – muru
    Sep 2, 2019 at 1:40
  • 1
    Good call. This is what I ended up using: find ./* -type f -print0|xargs -0 -P 16 -I % opusenc --bitrate 256 % %.opus Sep 2, 2019 at 3:52
  • 1
    Nice! You can post that as an answer.
    – muru
    Sep 2, 2019 at 4:54
  • 2
    No need for the ./*. A dot would be enough, and that would not blow up if you have many names in the current directory. I would also suggest a -name test to get names you want and not everything (and -type f to avoid running flac (or opusenc) on directories). If you have one single directory, you shouldn't need find at all, and if you're using bash or zsh, you could use ** instead of find.
    – Kusalananda
    Sep 2, 2019 at 6:57

1 Answer 1

8

You can set the number of workers if you use xargs

find . -type f -print0|xargs -0 -P 16 -I % opusenc --bitrate 256 % %.opus

xargs -P 16

(since I struggled to find more details online or understand the man page section)

It does your command on 16, or the number you pass to -P, of the inputs at a time.


For my command I was trying to convert an organized folders of cd music files [artist/album/music file] to opus. This was a copied folder so I wanted to delete the originals. So a better command than the one I used would be

find . -type f -print0|xargs -0 -P 16 -I % sh -c 'opusenc --bitrate 256 % %.opus && rm %'

After conversion the original file is removed.

3
  • 2
    a bit late, but for your "better" command consider changing the semicolon to && so you don't go around removing source files that failed to convert!
    – Asmadeus
    Jun 30, 2021 at 6:22
  • This is great. Is there a way to also add stdout redirection with % replacement ? When I add >%.log.txt at the end of the command, the % is not replaced there. The output tries to go to "%.log.txt" and only the last process wins. I could use >> but that would get all lines mixed up since the processes are going in parallel. Sep 25, 2021 at 4:48
  • 1
    @kalyanswaroop if you put the stdout redirection on the end of the opusenc command like this: find . -type f -print0|xargs -0 -P 16 -I % sh -c 'opusenc --bitrate 256 % %.opus > %.log.txt && rm %' I suspect you may also want stderr though; use find . -type f -print0|xargs -0 -P 16 -I % sh -c 'opusenc --bitrate 256 % %.opus > %.log.txt 2>&1 && rm %' to do that Sep 29, 2021 at 22:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .