How can I use sem to run a shell command in parallel in a for loop?

Question

This isn't speeding anything up for me:

domains=()

for i in `seq 1 100`; do
    echo $i
    word=`sem "xidel -s 'https://www.thisworddoesnotexist.com/' -e '//div[@id="definition-word"]/text()'"`
    len=${#word}
    if [ "$len" -lt "8" ]; then
        word=`echo $word | sed 's/[^a-z A-Z]//g'`
        domains+=(${word}.com)
    fi
done
sem --wait

printf '%s\n' "${domains[@]}"

I'm trying to run the for loop body in parallel with sem.

Answer 1

You are looking for parset:

myfunc() {
    word=$(xidel -s 'https://www.thisworddoesnotexist.com/' -e '//div[@id="definition-word"]/text()')
    len=${#word}
    if [ "$len" -lt "8" ]; then
        word=`echo $word | sed 's/[^a-z A-Z]//g'`
        echo ${word}.com
    fi
}
export -f myfunc

parset domains -j 30 myfunc ::: {1..100}
printf '%s\n' "${domains[@]}"

parset is part of GNU Parallel.

Answer 2

You need to tell sem how many jobs to run in parallel by specifying an appropriate --jobs flag, which is only 1 by default:

--jobs N
-j N
--max-procs N
-P N
    Run up to N commands in parallel. Default is 1 thus acting like a mutex.

However, sem won't return the data from xidel. It would be easiest to extract the body of the loop into its own script that writes its output into a file and then only run

domains=()

for i in $(seq 1 100); do
    echo $i
    sem --jobs=<N> script.sh "$i"
done
sem --wait
cat <some directory>/script_output_*

If your requirements allow that.