Say I have the following file:

A 1
B 2
CC 33

And I want to create a file containing every combination of two from the previous file, like this:

AA 11
AB 12
ACC 133
BA 21
BB 22
BCC 233
CCA 331
CCB 332
CCCC 3333

Can this be done with bash, for an arbitrary file? Each entry can consist of any characters except for newline and space. UTF-8 characters will be included in some entries.

I don't care about order.


You can try this with awk reading file twice:

awk 'NR == FNR { m[$1] = $2; next; } { for (i in m) { print $1 i, $2 m[i]; } }' file file
| improve this answer | |

You can do this entirely within the shell:

while read -r f1 f2
    while read -r f3 f4
        printf "%s %s\n" "$f1$f3" "$f2$f4"
    done < your_file
done < your_file

You say, "Each entry can consist of any characters except for newline and space."  If you really mean that entries can contain tab characters, say IFS=" " read instead of read (both times).

The “fine print”:

A command like read f1 f2 will read the first “word” on the line of input into variable f1 and the rest of the line into f2.  For example, the input The quick brown fox would result in f1="The" and f2="quick brown fox".  If you’re sure that your file will never have three (or more) columns (i.e., never have more than two words on any line), then there’s nothing to worry about.  If you’re happy with everything that’s not part of the first word being treated as part of the second word, then the above code should be OK. 

But, if you want The quick brown fox to be treated as f1="The" and f2="quick", with brown fox being discarded (ignored), then add a third variable to each of the read commands.  E.g., f1 f2 would become f1 f2 x; this will result in f1="The", f2="quick", and x="brown fox".  Simply by not using $x, we discard the input after the second word.  The second read can similarly be changed to read -r f3 f4 x — since we’re not using $x, it doesn’t matter if we overwrite it.  If you’d rather use a different throwaway variable — e.g., … f3 f4 y — that’s OK, too.

The read command, by default, treats the backslash (\) character specially.  Basically, backslash followed by any other character merge into a special version of the second character.  Thus, \C\C would be read as CC.  But, more importantly, backslash followed by space is not treated as a word separator, and backslash followed by newline (i.e., a backslash at the end of a line) is not treated as a line separator/terminator.  When we invoke read with the -r option, that stops, and backslash becomes an ordinary character.  Here are some practical examples of the differences:

            Without -r (default)                  __ With -r __
_Input_         f1        f2                      f1         f2
A\B\\C          AB\C                              A\B\\C
D\ E F          D E       F                       D\         E F    (or f2="E" and x="F")
G\          (this doesn’t count as a line)        G\
H               GH                                H

So I’ve added -r flags to my first version of my answer.  If you want to be able to handle D\ E as a single word, don’t use -r.

| improve this answer | |
  • It depends on both files synchronizing exactly - an additional blank line in either file during operation breaks this. If you want to treat variable length records like fixed-length records - which is what you do you compress delimiters - you should try to block them out first somehow - maybe conv=sync,block them with dd, for example. I also think xargs would be better suited here - you can get more specific about record length/size distributions, which is not to mention it would probably be a lot faster. But I would use paste - this is its job. – mikeserv May 16 '15 at 21:32

To preserve the order and avoid reading the file twice, I'd do:

awk '{f1[NR] = $1; f2[NR] = $2}
     END {
       for (i = 1; i <= NR; i++)
        for (j = 1; j <= NR; j++)
          print f1[i]f1[j], f2[i]f2[j]
     }' file

Now, that stores the whole file's content in memory before processing it (like in @taliezin's approach). If you'd rather not do that, then you'd need to read the file as many times as the file has lines like in @G-Man's approach. But using awk instead of sh/bash (which is not designed for that) would be a lot more efficient:

 awk '{f1=$1; f2=$2
       while ((getline < "file") > 0) print f1$1, f2$2
       close("file")}' file
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.