How to concatenate lines of several associated files into one line, and append that to an output file

Question

I have several file named BC**-tmp1.tsv that are the first iteration of a blast output and other file named BC**-tmp2.tsvthat are the second iteration.

Example of file BC02-tmp1.tsv (separator : \t) :

BC02    Aaa 2712    94  0   99.073  2053209 CP023507.1  1597    A
BC02    Bbb 2712    94  0   99.073  2053209 CP023507.1  1597    B
BC02    Ccc 2712    94  0   99.073  2053209 CP023507.1  1597    C
BC02    Ddd 2712    94  0   99.073  2053209 CP023507.1  1597    D

Example of file BC02-tmp2.tsv (separator : \t) :

BC02    Eee 2712    94  0   99.073  2053209 CP023507.1  1597    E
BC02    Fff 2712    94  0   99.073  2053209 CP023507.1  1597    F
BC02    Ggg 2712    94  0   99.073  2053209 CP023507.1  1597    G
BC02    Hhh 2712    94  0   99.073  2053209 CP023507.1  1597    H

My goal is to concatenate all those file by pair (iteration 1 + iteration 2) in a specific way.

Example of result with the example BC02 in the final file:

BC02    Aaa 2712    94  0   99.073  2053209 CP023507.1  1597    A   B   C   BC02    Eee 2712    94  0   99.073  2053209 CP023507.1  1597    E   F   G

So to be more precise I want to print (on the same line) the first line of the BC**-tmp1.tsvfile, then the last column of the second line, then the last column of the third line, then the first line of the BC**-tmp2.tsvfile, then the last column of the second line, then the last column of the third line. And that for each pair of Barcodes.

Note: the second iteration file is not always present.

So far I managed to gather the associated files in a shell for loop, but I don't know how to do the rest:

touch template.tsv

for bla in *-tmp1.tsv; do
r="$(basename -s "-tmp1.tsv" $bla)"
awk 'FNR==1' $bla >> template.tsv
awk 'FNR==1' $r-tmp2.tsv >> template.tsv;
done

Do you know how to do that ?

---

EDIT simpler input/output:

Input:

$ head BC02-tmp*.tsv
==> BC02-tmp1.tsv <==
a       b       B
a       c       C
a       d       D
a       e       E

==> BC02-tmp2.tsv <==
a       w       W
a       x       X
a       y       Y
a       z       Z

Output:

a       b       B       C       D       a       w       W       X       Y

Answer 1

Assuming that your input doesn't contain fringe cases, the following shell loop in connection to an awk program should do:

for f in BC*-tmp1.tsv
do
    f2="${f/%tmp1.tsv/tmp2.tsv}"
    if [[ ! -f $f2 ]]; then f2=""; fi
    awk 'BEGIN{FS=OFS="\t"}
         FNR==1{for (i=1;i<NF;i++) printf "%s%s",(NR==FNR&&i==1?"":OFS),$i}
         FNR<=3{printf "%s%s",OFS,$NF}
         END{printf "%s",ORS}' "$f" "$f2" >> template.tsv
done

This will loop over all tmp1.tsv files and generate the corresponding filename for the tmp2.tsv file. If the second file turns out not to exist, the filename will be set to the empty string.

It will then call an awk program with both associate TSV files, which will print - all on the same line

• all fields, excluding the last one, of the first line of each input file (but preceded by an additional OFS in case of the second input file, characterized by FNR, the per-file line counter, no longer being equal to NR, the global line counter),
• the last field of each line until line 3,
• when no input file remains, a closing record separator (defaults to newline)

and appends the output to template.tsv. This will also work if the second template file doesn't exist, because the empty string token will not be recognized as input file by awk in the first place, so the END section printing the newline will be reached after the first file already.