GNU parallel bash

Question

I am learning GNU parallel and I wonder it this makes sense:

bash:

    IFS=" "
while read field || [ -n "$field" ]; do
    targets_array+=("$field")
    
done </location/targets
 
parallel  -Dall bash myScript.sh ::: $targets_array

I wonder if it makes sense because my output seem to stop at some point... I have 30.000 targets that I scan with myScript.sh then I update info about them in DB also using myScript.sh

I tried to use some options but I could not make it work: like writing to a logfile from the performance point of view, does it make sense to run one target at the time?

Answer 1

$targets_array is equivalent to ${targets_array[0]}. To get all elements you need ${targets_array[@]}. And you should quote right.

So it could be:

parallel … ::: "${targets_array[@]}"               # but don't

parallel is an external command. If the array is large enough then you will hit argument list too long. Use this instead:

printf '%s\n' "${targets_array[@]}" | parallel …   # still not the best

It will work better because in Bash printf is a builtin and therefore everything before | is handled internally by Bash.

I notice you didn't use read -r (I assume it was an educated decision), so a backslash-newline pair (if any) in /location/targets can result in a newline character actually in some array element. Therefore separating with newlines while passing data to parallel may be a bug. Separate with null bytes:

printf '%s\0' "${targets_array[@]}" | parallel -0 …

Hopefully /location/targets does not contain null bytes. If it does then they won't get to the array in the first place.