1

This may be a couple questions in one, but this problem has me stuck. I have a text file with groupings of four lines separated by an empty line. The first line of every grouping begins with a '>' symbol. I want to delete all instances of groupings with identical first lines, except for the first instance of the grouping. First thing that comes to mind is using sed but cant think of a good way to implement. Would be grateful for any help!

input:

>abc1234.54321
linea1
lineaa1
lineaaa1

>def56789034
linea1
lineaa1
lineaaa1

>abc1234.54321
linea2
lineaa2
lineaaa2

>def56789034
linea2
lineaa2
lineaaa2

intended output:

>abc1234.54321
linea1
lineaa1
lineaaa1

>def56789034
linea1
lineaa1
lineaaa1
1

If you don't mind a possible additional trailing blank line, then Awk in paragrapgh mode will get you close:

$ awk -vRS= '!seen[$1]++' ORS='\n\n' input
>abc1234.54321
linea1
lineaa1
lineaaa1

>def56789034
linea1
lineaa1
lineaaa1

Unsetting the record separator (RS=) causes Awk to treat each group of blank-line-separated lines as a single record; then we use the value of an associative array keyed on the first field $1 to keep track of whether we have seen a record with this key before - if not (i.e. !seen[$1] is true) then print it.

  • Would you mind explaining how this works? – Gandhar Mahadeshwar Jun 20 at 21:46
  • @GandharMahadeshwar I have added a brief explanation – steeldriver Jun 20 at 21:50
  • 1
    Nice. I would also add -F'\n' – glenn jackman Jun 20 at 23:20
  • Puzzled by the blend of -v and post-script variables. – glenn jackman Jun 20 at 23:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.