Group files that share same header then concatenate them

Question

I'm using Bash on Ubuntu and my issue is the following:

I have dozens of .TXT files in a certain folder.

As far as I know, all of them have a header (I believe every header holds on 1 line but not 100% sure). However, they do not necessary share the same header. Five files might have the same header while another file might have a unique header.

Ultimately, what I would like to do is concatenate the files that share the same header. The answers to the following question (Concatenate multiple files with same header) explain how to concatenate multiple files with the same header, however in my case I would first need to group the files sharing the same header before concatenating them (and only keep the header of first file among said groups).

Any ideas are welcome :) Thank you !

Answer 1

awk '
  FNR==1{
    if (!($0 in h)){file=h[$0]=i++}
    else{file=h[$0];next}
  }
  {print >> (file)}
' *.txt

If awk is in a first line of a file:

• If the header is not present in the header array h, then set the number i++ (initially zero) as the file name and also put it in the key $0 of the array.
• Otherwise (if the header is already in the array h), fetch the file name from the array and read the next line.

Finally, the line is printed to the corresponding file.

---

But I get "too many open files" or something of the sort.

GNU awk handles opening and closing files on demand to circumvent that, but other awks may not do it. In such case, go for

awk '
  FNR==1{
    if (!($0 in h)||file!=h[$0]){close(file)}
    if (!($0 in h)){file=h[$0]=i++}
    else{file=h[$0];next}
  }
  {print >> (file)}
' *.txt

Bear in mind this can be slower.