How to process multiple files with pandoc?

Question

files:

$ ls a.md b.md c.md d.md e.md

Command: pandoc file.md -f markdown file.pdf

How would I parallely process two pandoc instances simulatneously? Possibly with xargs or parallel.

It would work like

Iteration/ cmd 1 / cmd 2

1 / pandoc a.md -f markdown a.pdf / pandoc b.md -f markdown b.pdf

2 / pandoc c.md -f markdown c.pdf / pandoc d.md -f markdown d.pdf

3 / pandoc e.md -f markdown e.pdf / pandoc f.md -f markdown f.pdf

4 / pandoc g.md -f markdown g.pdf / pandoc h.md -f markdown h.pdf

The files are randomly named.

Answer 1

This might work:

parallel pandoc {} -f markdown {.}.pdf ::: *.md

If you do not want to run one job per CPU thread, but want to force 2 in parallel:

parallel -j2 pandoc {} -f markdown {.}.pdf ::: *.md

You may want to spend 15 minutes on reading chapter 1+2 of the GNU Parallel book: http://www.lulu.com/shop/ole-tange/gnu-parallel-2018/paperback/product-23558902.html (Printed), https://doi.org/10.5281/zenodo.1146014 (Online).

Answer 2

Crudely,

#!/bin/sh

set -- *.md
while [ $# -gt 0 ]
do
  pandoc "${1} -f markdown -o ${1%.md}.pdf" &
  shift
  if [ $# -gt 0 ]
  then
    pandoc "${1} -f markdown -o ${1%.md}.pdf" &
    shift
  fi
  wait
done

With xargs:

find . -type f -name '*.md' -print0 | xargs -0 -n2 -P2 -I{} pandoc {} -f markdown -o {}.pdf

you would have to rename them afterwards, as the above would result in files named a.md.pdf, b.md.pdf, etc. Note that to be safe with filenames, we're asking find to print null-separated filenames and asking xargs to read in null-separated input. Rename the files with:

for f in ./*.md.pdf; do mv -- "${f}" "${f%.md.pdf}.pdf"; done