2

I'm using Bash on Ubuntu and my issue is the following:

I have dozens of .TXT files in a certain folder.

As far as I know, all of them have a header (I believe every header holds on 1 line but not 100% sure). However, they do not necessary share the same header. Five files might have the same header while another file might have a unique header.

Ultimately, what I would like to do is concatenate the files that share the same header. The answers to the following question (Concatenate multiple files with same header) explain how to concatenate multiple files with the same header, however in my case I would first need to group the files sharing the same header before concatenating them (and only keep the header of first file among said groups).

Any ideas are welcome :) Thank you !

2
awk '
  FNR==1{
    if (!($0 in h)){file=h[$0]=i++}
    else{file=h[$0];next}
  }
  {print >> (file)}
' *.txt

If awk is in a first line of a file:

  • If the header is not present in the header array h, then set the number i++ (initially zero) as the file name and also put it in the key $0 of the array.
  • Otherwise (if the header is already in the array h), fetch the file name from the array and read the next line.

Finally, the line is printed to the corresponding file.


But I get "too many open files" or something of the sort.

GNU awk handles opening and closing files on demand to circumvent that, but other awks may not do it. In such case, go for

awk '
  FNR==1{
    if (!($0 in h)||file!=h[$0]){close(file)}
    if (!($0 in h)){file=h[$0]=i++}
    else{file=h[$0];next}
  }
  {print >> (file)}
' *.txt

Bear in mind this can be slower.

8
  • There are many columns in these tables so the headers are very long. As a result, I get the error awk: .. fatal: can't redirect to... : File name too long . That is since the output file name is determined as the header. – Alex Aug 3 '20 at 10:30
  • @Alex Check out the edit. Hope it helps. Otherwise, please show us at least one fragment of one file of yours. – Quasímodo Aug 3 '20 at 11:59
  • I know it doesn't help but I am not allowed to show you any fragment of file... So I tried running the edited version (I thought I already had but I hadn't, my bad) and after 10 minutes it created 7 new files called 3,4,5,6,7,8 and 9. I guess it means there were 7 distinct headers initially – Alex Aug 3 '20 at 12:19
  • @Alex You could come up with a file that resembled yours, but that's OK. Are you sure there are no 0, 1 and 2 files too? The new files are supposed to begin from 0. – Quasímodo Aug 3 '20 at 12:33
  • I wrote other comments but just deleted them because nevermind I just realized that there are files 0, 1 and 2 and moreover not 9 but 18 files were created. I'm not making sure the number of lines add up – Alex Aug 3 '20 at 13:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.