I have the following, working BASH script, which I run on a Mac Pro 2010/Mojarve OS:



# count up wav files
cnt=$(find /hummdinger/LoCI/LoCI_orig/VO/WAV_Processed/ -name "*.wav" | wc -l)
echo "there are $cnt .wav voice samples."

# go through and run rhubarb on them
for f in $(find /hummdinger/LoCI/LoCI_orig/VO/WAV_Processed/ -name "*.wav")
    echo "$c of $cnt";
    f=$(basename "$f" .wav)
    /hummdinger/LoCI/LoCI_orig/TSV/rhubarb-lip-sync-1.10.0-osx/rhubarb /hummdinger/LoCI/LoCI_orig/VO/WAV_Processed/"$f".wav -o /hummdinger/LoCI/LoCI_orig/LOCI_GAME_FILES/Compiled/Windows/sync/"$f".tsv

It takes a list of WAV files, goes through each, scans the file and then produces output and stores the generated TSV files somewhere else. The point of 'rhubarb' is to produce lip-sync information from a recording (the WAV files). etc. etc. blah blah.

The one problem with this script is it takes ~10-12 HOURS to run over about 3,000 wav files. On my crappier, non-ECC ram, one-time-it-corrupted-the-whole-lot-and-I-made-a-vow-to-never-use-it-again Mac Mini 2018, it took about 3 hours.

But this is a Mac Pro, meaning though it's old (2010), but it's very reliable and has 12x Xeons. This is pretty low intensity work, so I'm missing out on that extra juice by making it single processor. I just want to get this script working with 10-15-30 threads, and hopefully this will speed it up and it'll be done in an hour or less; not most of the day.

My thoughts are: divide the directory of WAVs into groups of (total_files/15), put these listings in file1-15.txt, then read each one back and process it in 15 separate threads. But that's about as far as I've gotten :P

Can anyone help with making this a muti-process script? I'm an amateur and made this script with help from reddit.

  • 1
    Have you looked in to other options as well, such as giving the process more CPU time (changing it's niceness, priority, etc)? Multi-threaded by itself will not help much unless you can run the threads in parallel/concurrently in different CPUs. And even then, your "bottleneck" will be the system bus and keeping each CPU fed with instructions and data to work with (this is why I suggest a higher priority--the kernel will give it more CPU time)
    – C. M.
    Jun 22 '21 at 8:39
  • 1
    Can you adapt the process to use GNU parallel e.g.?
    – AdminBee
    Jun 23 '21 at 14:11
  • Have you tried splitting your files into TWO groups and starting 2 processes (with suitably modified file names) to check if you get any speed improvement at all? If not, then multithreading will not help you. For example, how much I/O is happening? Can your disk storage cope? Jun 23 '21 at 18:15

With GNU Parallel you can do something like this:


find /hummdinger/LoCI/LoCI_orig/VO/WAV_Processed/ -name "*.wav" |
  parallel $rhubarb {} -o {.}.tsv

Or (if you really need the output in a different dir):

find /hummdinger/LoCI/LoCI_orig/VO/WAV_Processed/ -name "*.wav" |
  parallel $rhubarb {} -o /hummdinger/LoCI/LoCI_orig/LOCI_GAME_FILES/Compiled/Windows/sync/{/.}.tsv

Write your script so that it iterates over its arguments. For example:



for fn in "$@"; do
    bn=$(basename "$fn" .wav)
    "$rhubarb" "$fn" -o "$outdir/$bn.tsv"

Save this as, e.g., myscript1.sh and make it executable with chmod +x myscript1.sh.

You can run this directly, but it will process each file sequentially. Instead, you want to run it with GNU parallel or xargs -P. E.g. with a wrapper script like the following, which divides the number of files to process by the number of cores you have.

Note that depending on exactly what rhubarb does, this is likely to be more of an I/O-bound task than CPU-bound so adding too many cores isn't going to help - in fact, it will probably slow things down as there will be too much contention for disk I/O...especially if you are running this on a HDD rather than an SSD.

You might want to hard-code something like cores=4 or cores=8 in the script below rather than use lscpu | awk ... as I have (I wrote it like that because I'm running a threadripper 1950x with 16 cores and 32 threads....and I didn't want to run 32 jobs in parallel. And also as an example of how you can extract useful info from lscpu).

Also recommended: if you have more than one drive, try to arrange things so that the directory you read the .wav files from is on one drive and the directory you write the .tsv files to is on another. This will eliminate I/O contention between reading and writing the files. If the .tsv files aren't huge, write them to a temporary directory on a tmpfs ramdisk and move them to their final location at the end of the script.



cores=$(lscpu | awk -F': +' '/^CPU\(s\):/ {cpus=$2};
                             /^Thread\(s\) per core:/ {tpc=$2};
                             END { print int(cpus / tpc) }')

count=$(find "$wavdir" -type f -name "*.wav" -print0 |
          perl -0ne '$c++;END{print $c}')

let files_per_thread=count/cores

find "$wavdir" -type f -name "*.wav" -print0 |
    xargs -0 -r -L "$files_per_thread" -P "$cores" /path/to/myscript1.sh

save this as, e.g., myscript2.sh and make it executable with chmod +x myscript2.sh.

This is the script you run from the command line or cron, or whatever. It, in turn, uses xargs to run multiple instances of myscript1.sh in parallel.

Run it like:

./myscript2.sh /hummdinger/LoCI/LoCI_orig/VO/WAV_Processed/

BTW, this uses NUL as the separator between filenames so is safe to use with any filename (using newline as the filename separator is not safe because newline is a valid character within a filename).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.