2

files:

$ ls a.md b.md c.md d.md e.md

Command: pandoc file.md -f markdown file.pdf

How would I parallely process two pandoc instances simulatneously? Possibly with xargs or parallel.

It would work like

Iteration/ cmd 1 / cmd 2

1 / pandoc a.md -f markdown a.pdf / pandoc b.md -f markdown b.pdf

2 / pandoc c.md -f markdown c.pdf / pandoc d.md -f markdown d.pdf

3 / pandoc e.md -f markdown e.pdf / pandoc f.md -f markdown f.pdf

4 / pandoc g.md -f markdown g.pdf / pandoc h.md -f markdown h.pdf

The files are randomly named.

1

Crudely,

#!/bin/sh

set -- *.md
while [ $# -gt 0 ]
do
  pandoc "${1} -f markdown ${1%.md}.pdf" &
  shift
  if [ $# -gt 0 ]
  then
    pandoc "${1} -f markdown ${1%.md}.pdf" &
    shift
  fi
  wait
done

With xargs:

find . -type f -name '*.md' -print0 | xargs -0 -n2 -P2 -I{} pandoc {} -f markdown {}.pdf

you would have to rename them afterwards, as the above would result in files named a.md.pdf, b.md.pdf, etc. Note that to be safe with filenames, we're asking find to print null-separated filenames and asking xargs to read in null-separated input. Rename the files with:

for f in ./*.md.pdf; do mv -- "${f}" "${f%.md.pdf}.pdf"; done
  • I added a -P2 to the command I ran. – Latex_xetaL Oct 28 '18 at 17:10
  • I edited -P2 into the answer to match; thanks! – Jeff Schaller Oct 28 '18 at 18:48
2

This might work:

parallel pandoc {} -f markdown {.}.pdf ::: *.md

If you do not want to run one job per CPU thread, but want to force 2 in parallel:

parallel -j2 pandoc {} -f markdown {.}.pdf ::: *.md

You may want to spend 15 minutes on reading chapter 1+2 of the GNU Parallel book: http://www.lulu.com/shop/ole-tange/gnu-parallel-2018/paperback/product-23558902.html (Printed), https://doi.org/10.5281/zenodo.1146014 (Online).

  • if I'm understanding the online doc correctly, the OP would want --jobs 2 ? – Jeff Schaller Oct 30 '18 at 0:10
  • Neat, thanks. But @JeffSchaller is correct – Latex_xetaL Oct 30 '18 at 5:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.