2

I am learning GNU parallel and I wonder it this makes sense:

bash:

    IFS=" "
while read field || [ -n "$field" ]; do
    targets_array+=("$field")
    
done </location/targets
 
parallel  -Dall bash myScript.sh ::: $targets_array

I wonder if it makes sense because my output seem to stop at some point... I have 30.000 targets that I scan with myScript.sh then I update info about them in DB also using myScript.sh

I tried to use some options but I could not make it work: like writing to a logfile from the performance point of view, does it make sense to run one target at the time?

1 Answer 1

4

$targets_array is equivalent to ${targets_array[0]}. To get all elements you need ${targets_array[@]}. And you should quote right.

So it could be:

parallel … ::: "${targets_array[@]}"               # but don't

parallel is an external command. If the array is large enough then you will hit argument list too long. Use this instead:

printf '%s\n' "${targets_array[@]}" | parallel …   # still not the best

It will work better because in Bash printf is a builtin and therefore everything before | is handled internally by Bash.

I notice you didn't use read -r (I assume it was an educated decision), so a backslash-newline pair (if any) in /location/targets can result in a newline character actually in some array element. Therefore separating with newlines while passing data to parallel may be a bug. Separate with null bytes:

printf '%s\0' "${targets_array[@]}" | parallel -0 …

Hopefully /location/targets does not contain null bytes. If it does then they won't get to the array in the first place.

6
  • 2
    I don't really see the point of the array at all actually. Why not just read targets directly with parallel?
    – Kusalananda
    Commented Dec 2, 2020 at 8:11
  • 1
    @Kusalananda Because I assume read without -r was an educated decision. I'm not aware of GNU parallel's ability to handle backslash equivalently. Can it do this? Commented Dec 2, 2020 at 8:15
  • 1
    I assumed that this was an omission in the question. Setting IFS to a space character indicates that they either don't know what this does (and that omitting -r was a mistake), or that they explicitly want to trim flanking spaces from the input data (in which case omitting -r may have been on purpose also). Unfortunately, the question does not tell.
    – Kusalananda
    Commented Dec 2, 2020 at 8:31
  • @Kusalananda The answer is now community wiki. If you think it should elaborate on the lack of -r or provide a command to read targets directly with parallel then please improve it. Unfortunately I have no time for this now. Commented Dec 2, 2020 at 8:39
  • @Kusalananda It looks like the omission of -r was a mistake, I tried to set the field separator as a space and read target after target separated by space from the file.
    – dwt.bar
    Commented Dec 2, 2020 at 17:01

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .