1

I have a big text file containing a million lines. I would like to find identical lines that match my specific text and leave the first occurrence intact. Any ideas?

So the algorithm should roughly be as follows:

For all lines in text file
   Find duplicate lines
   If duplicated line contains our text
      Remove all these lines except the first one

For example, we are looking for the duplicated lines with text “Word”.
Example of the file:

Test1 Test2 Test3 Test4 Test5
Test1 Test2 Test3 Test4 Test5
Word1 Word2 Word3 Word4 Word5
Test1 Test2 Test3 Test4 Test5
Word1 Word2 Word3 Word4 Word5

The output file:

Test1 Test2 Test3 Test4 Test5
Test1 Test2 Test3 Test4 Test5
Word1 Word2 Word3 Word4 Word5
Test1 Test2 Test3 Test4 Test5
3
  • 1
    It is not clear to me what you want. Are you trying to edit the file and remove all but the first of lines containing your specific text? An example of the file before and after would be useful, even if you have to make up a fake one because you cannot post your real data. – bitinerant Jan 13 at 5:04
  • @bitinerant Sure! Added ;) – nmzik Jan 13 at 6:14
  • Am I right in assuming that you want the equivalent of grep PATTERN | uniq, with the caveats that non-matches are also printed, only pattern matches are passed to uniq, and order is preserved? – l0b0 Jan 13 at 7:03
3

One simple way can be:

awk '!/Word/ || !a[/Word/]++' file

awk 'm=!/Word/ || !a[!m]++'

which translates into : print all lines not having Word OR if they do have it, then print only the first line having it and inhibit the rest.

Note: you may need to tighten what is Word which I leave it to you as an exercise.

0
2

If you want to do a partial regexp comparison (see https://stackoverflow.com/questions/65621325/how-do-i-find-the-text-that-matches-a-pattern):

$ awk -v regexp='Word' '!(($0 ~ regexp) && seen[$0]++)' file
Test1 Test2 Test3 Test4 Test5
Test1 Test2 Test3 Test4 Test5
Word1 Word2 Word3 Word4 Word5
Test1 Test2 Test3 Test4 Test5

or if you want a partial string comparison:

$ awk -v string='Word' '!(index($0,string) && seen[$0]++)' file
Test1 Test2 Test3 Test4 Test5
Test1 Test2 Test3 Test4 Test5
Word1 Word2 Word3 Word4 Word5
Test1 Test2 Test3 Test4 Test5

If you wanted some other kind of matching then please edit your question to clarify.

1

You can try this:

sed '0,/Word/b;/Word/d' file

Delete the lines that match "Word":

/Word/d 

From the beginning of file to the first line matching "Word", branch to the end of script -- that is, skip over the deletion command:

0,/Word/b

From this stackoverflow post.

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.