Remove duplicated lines containing specific text

Question

I have a big text file containing a million lines. I would like to find identical lines that match my specific text and leave the first occurrence intact. Any ideas?

So the algorithm should roughly be as follows:

For all lines in text file
   Find duplicate lines
   If duplicated line contains our text
      Remove all these lines except the first one

For example, we are looking for the duplicated lines with text “Word”.
Example of the file:

Test1 Test2 Test3 Test4 Test5
Test1 Test2 Test3 Test4 Test5
Word1 Word2 Word3 Word4 Word5
Test1 Test2 Test3 Test4 Test5
Word1 Word2 Word3 Word4 Word5

The output file:

Test1 Test2 Test3 Test4 Test5
Test1 Test2 Test3 Test4 Test5
Word1 Word2 Word3 Word4 Word5
Test1 Test2 Test3 Test4 Test5

Answer 1

One simple way can be:

awk '!/Word/ || !a[/Word/]++' file

awk 'm=!/Word/ || !a[!m]++'

which translates into : print all lines not having Word OR if they do have it, then print only the first line having it and inhibit the rest.

Note: you may need to tighten what is Word which I leave it to you as an exercise.

Answer 2

If you want to do a partial regexp comparison (see https://stackoverflow.com/questions/65621325/how-do-i-find-the-text-that-matches-a-pattern):

$ awk -v regexp='Word' '!(($0 ~ regexp) && seen[$0]++)' file
Test1 Test2 Test3 Test4 Test5
Test1 Test2 Test3 Test4 Test5
Word1 Word2 Word3 Word4 Word5
Test1 Test2 Test3 Test4 Test5

or if you want a partial string comparison:

$ awk -v string='Word' '!(index($0,string) && seen[$0]++)' file
Test1 Test2 Test3 Test4 Test5
Test1 Test2 Test3 Test4 Test5
Word1 Word2 Word3 Word4 Word5
Test1 Test2 Test3 Test4 Test5

If you wanted some other kind of matching then please edit your question to clarify.

Answer 3

You can try this:

sed '0,/Word/b;/Word/d' file

---

Delete the lines that match "Word":

/Word/d 

---

From the beginning of file to the first line matching "Word", branch to the end of script -- that is, skip over the deletion command:

0,/Word/b

---

From this stackoverflow post.