4

I'm running something like this:

find . -maxdepth 1 -type f -note -iname "*.gpg" | sort | while read file ; do
    echo "Encrypting $file..."
    gpg --trust-model always --recipient "me@myself.com" --output "$file.gpg" \
        --encrypt "$file" && rm "$file" 
done

This runs great, but it seems that GPG is not optimized to use multiple cores for an encryption operation. The files I'm encrypting are about 2GB in size and I have quite a bit of them. I'd like to be able to run X jobs in parallel to encrypt the files and then remove them. How can I do this, setting a limit to, say, 8 jobs at a time?

5

If you install the GNU Parallel tool you can make pretty easy work of what you're trying to accomplish:

$ find . -maxdepth 1 -type f -note -iname "*.gpg" | sort | \
      parallel --gnu -j 8 --workdir $PWD '                 \
         echo "Encrypting {}...";                          \
         gpg --trust-model always                          \
           --recipient "me@myself.com" --output "{}.gpg"   \
           --encrypt "{}" && rm "{}"                       \
      '

details

The above is taking the output of find and running it through to parallel, and running 8 at a time. Everywhere there's an occurrence of {} the filenames that are being passed through from find will replace the {} in those spots.

References

  • What's with the hanging apostrophe at the end? Is it missing a starting apostrophe? – Naftuli Kay Jun 24 '13 at 17:43
  • @TKKocheran - did that work for you? – slm Jun 24 '13 at 17:55
  • This is what I'm running, and this is the output I'm seeing. Not working yet :-| – Naftuli Kay Jun 24 '13 at 18:08
  • What if you move the sudo outside of the find? Make it a sudo -i find ... I don't think you can pipe to sudo in that manner. You could do this: echo "cmd1" | sudo bash. – slm Jun 24 '13 at 18:20
  • 1
    Looks like I got pwnt with this problem. – Naftuli Kay Jun 24 '13 at 18:32
4

You might want to look at gnu parallel and its --semaphore option. From the documentation:

--semaphore

Work as a counting semaphore. --semaphore will cause GNU parallel to start command in the background. When the number of simultaneous jobs is reached, GNU parallel will wait for one of these to complete before starting another command.

You use --jobs 8 to limit the number of jobs to 8. You can pipe the output of sort into parallel like you would do with xargs. sem is an alias for parallel --semaphore

0

I have written an easy-to-use Perl script that allows to control the maximum number commands that are run at the same time: https://github.com/matmu/parallelize_cmds

Might be of interest for you.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.