1

I am learning GNU parallel and I wonder it this makes sense:

bash:

    IFS=" "
while read field || [ -n "$field" ]; do
    targets_array+=("$field")
    
done </location/targets
 
parallel  -Dall bash myScript.sh ::: $targets_array

I wonder if it makes sense because my output seem to stop at some point... I have 30.000 targets that I scan with myScript.sh then I update info about them in DB also using myScript.sh

I tried to use some options but I could not make it work: like writing to a logfile from the performance point of view, does it make sense to run one target at the time?

3

$targets_array is equivalent to ${targets_array[0]}. To get all elements you need ${targets_array[@]}. And you should quote right.

So it could be:

parallel … ::: "${targets_array[@]}"               # but don't

parallel is an external command. If the array is large enough then you will hit argument list too long. Use this instead:

printf '%s\n' "${targets_array[@]}" | parallel …   # still not the best

It will work better because in Bash printf is a builtin and therefore everything before | is handled internally by Bash.

I notice you didn't use read -r (I assume it was an educated decision), so a backslash-newline pair (if any) in /location/targets can result in a newline character actually in some array element. Therefore separating with newlines while passing data to parallel may be a bug. Separate with null bytes:

printf '%s\0' "${targets_array[@]}" | parallel -0 …

Hopefully /location/targets does not contain null bytes. If it does then they won't get to the array in the first place.

6
  • 2
    I don't really see the point of the array at all actually. Why not just read targets directly with parallel? – Kusalananda Dec 2 '20 at 8:11
  • 1
    @Kusalananda Because I assume read without -r was an educated decision. I'm not aware of GNU parallel's ability to handle backslash equivalently. Can it do this? – Kamil Maciorowski Dec 2 '20 at 8:15
  • 1
    I assumed that this was an omission in the question. Setting IFS to a space character indicates that they either don't know what this does (and that omitting -r was a mistake), or that they explicitly want to trim flanking spaces from the input data (in which case omitting -r may have been on purpose also). Unfortunately, the question does not tell. – Kusalananda Dec 2 '20 at 8:31
  • @Kusalananda The answer is now community wiki. If you think it should elaborate on the lack of -r or provide a command to read targets directly with parallel then please improve it. Unfortunately I have no time for this now. – Kamil Maciorowski Dec 2 '20 at 8:39
  • @Kusalananda It looks like the omission of -r was a mistake, I tried to set the field separator as a space and read target after target separated by space from the file. – dwt.bar Dec 2 '20 at 17:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.