3

I have an input file, names.txt, with the 1 word per line:

apple  
abble  
aplle 

With my bash script I am trying to achieve the following output:

apple and apple  
apple and abble  
apple and aplle  
abble and apple  
abble and abble  
abble and aplle  
aplle and apple  
aplle and abble  
aplle and aplle

Here is my bash script

#!/usr/bin bash
readarray -t seqcol < names.txt

joiner () {
           val1=$1
           val2=$2
           echo "$val1 and $val2"
}

export -f joiner

parallel -j 20 '
    line=($(echo {}))
    for word in "${line[@]}"; do
            joiner "${line}" "${word}"
    done
' ::: "${seqcol[@]}"

but it is only outputting the following 3 lines comparing identical elements from the array

apple and apple  
abble and abble  
aplle and aplle

I have the script that uses while read line loop, but it is too slow (my actual datafile is has about 200k lines). That is why I want to use array elements and gnu parallel at the same to speed the process up.
I have tried different ways of accessing the array elements within the parallel ' ' command (by mainly modifying this loop - for word in "${line[@]}", or by supplying the array to parallel via printf '%s\n' "${seqcol[@]}") but they are either leading to errors or output blank lines.

I would appreciate any help!

0

2 Answers 2

2

parallel won't necessarily keep the right order of output.

Anyways, if you care about performance, don't use bash to work with text files. Instead use a dedicated tool, like awk or a programming language, e.g. python.

awk '
  { words[NR]=$0 }
  END {
    for (x in words){
      for (y in words) { printf "%s and %s\n",words[x],words[y] }
    }
  }' file
3
  • Thanks for quick response, @pLumo! The awk solution is much neater, but can I still pass a function into awk and will it be faster than gnu parallel or xargs alternatives? (As sometimes there is more than 1 function and they are more complex than just printing 2 strings with "and" in between)
    – duda13
    Apr 28, 2023 at 16:11
  • @duda13 the shell will almost always be the slowest option for this, so yes, you should avoid it whenever you can. Awk is a full programming language, so you can do anything you want to do, but no you can't export bash functions into awk, you would have to write an awk function instead.
    – terdon
    Apr 28, 2023 at 16:34
  • I am aware that shell is not the best solution, but I cannot use python or R in this case. And I am not that familiar with awk scripting, especially in the context of complex functions. I have functions in bash that work well, but I just need to parallelize the script to make them faster.
    – duda13
    Apr 28, 2023 at 17:58
0

GNU Parallel can generate all combinations of input sources.

In your case you simply use names.txt twice:

parallel -k echo {1} and {2} :::: names.txt names.txt

Or (if you really have an array):

readarray -t seqcol < names.txt
parallel -kj 20 echo {1} and {2} ::: "${seqcol[@]}" ::: "${seqcol[@]}"

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .