1

Given a file with paragraphs (group of lines separated by a blank line). It is guaranteed that the line breaks within equivalent paragraphs are conserved. Is there a way of using uniq to remove the duplicate paragraphs?

I suppose one could serialize each paragraph into a single line, perform uniq and deserialize the result, is there a better solution?

Input:

Paragraph1
continue
continue

...

Paragraph2
continue
continue

...

Paragraph1
continue
continue

...

Paragraph2
continue
continue

Desired output: only one Paragraph1 and Paragraph2.

A command line solution would be preferable.

2
$ awk -v RS= -v ORS='\n\n' '!seen[$0]++' file
Paragraph1
continue
continue

...

Paragraph2
continue
continue
| improve this answer | |
0

Looking at awk documentation it seems that I could use the following:

cat file.txt | awk 'BEGIN { RS = "\n\n+"; first = 1 } {if (!x[$0]++) { if (!first) {print ""}; print $0; first = 0} }'

Quick explanation:

  • We set the record separator RS to "\n\n+" to separate paragraphs.
  • The variable first is used to print a newline between records, alternatively you could print a newline after every $0, but then I didn't have a way of removing it for the last record.
  • The map x is used to count if the record has been seen before, and only if it hasn't been seen will we print the record.
  • If it is not the first record, we print a newline between records
  • Print the record.
  • Set first to False.
| improve this answer | |
  • Or awk -v RS= '! seen[$0]++ {print (NR == 1 ? "" : ORS) $0}' file.txt. Logically similar, possibly a bit more idiomatic. – fra-san Sep 17 at 9:49
  • Thanks that is indeed shorter, and I get to read about about ORS :) – Frankie Y. Liu Sep 17 at 22:34
  • That would require GNU awk for multi-char RS. Please also see porkmail.org/era/unix/award.html. – Ed Morton Sep 18 at 2:34
  • @EdMorton GNU awk is not the only awk which supports multi-char RS. First was mawk since a couple of DECADES ago, but now "bwk" (updated version of the original awk) does support it too. It's not the first time I see this bit of misinformation thrown around, it's high time to give up on it once for all. – user414777 Sep 18 at 2:48
  • @user414777 Yeah, you're right, I need to start saying something about it being non-POSIX rather than gawk-only. – Ed Morton Sep 18 at 12:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.