I have two files, let's call them 123.txt and 789.txt. 123.txt is 2.5M lines long, and 789.txt is 65M lines long. Is there any way to use grep or similar to keep any lines from 789.txt that contain lines from 123.txt?

There will be a max of one duplicate per line in 789.txt, and the duplicate text will be at the beginning of the line. I'm totally stuck on this, and couldn't find any info online, so I don't really have anything to start with. It will be running on a server, so I don't mind it taking a while (which I know it will)

  • 123.txt:

  • 789.txt:

  • Desired output:


You can do this very easily using grep:

$ grep -Ff 123.txt 789.txt

The command above will print all lines from file 789.txt that contain any of the lines from 123.txt. The -f means "read the patterns to search from this file" and the -F tells grep to treat the search patterns as strings and not its default regular expressions.

This will not work if the lines of 123.txt contain trailing spaces, grep will treat the spaces as part of the pattern to look for an will not match if it occurs within a word. For example, the pattern foo (note the trailing space) will not match foobar. To remove trailing spaces from your file, run this command:

$ sed 's/ *$//' 123.txt > new_file

Then use the new_file to grep:

$ grep -Ff new_file 789.txt

You can also do this without a new file, using the i flag:

$ sed -i.bak 's/ *$//' 123.txt

This will change file 123.txt and keep a copy of the original called 123.txt.bak.

(Note that this form of the -i flag to sed assumes you have GNU sed; for BSD sed use -i .bak with a space in between.)

  • I don't think the first option would work, regardless of the OP's case: it would print lines in 123.txt that aren't in 789.txt as well as lines in 789.txt that appear in 123.txt (they will only be printed once, but printed nonetheless). – Joseph R. Aug 1 '13 at 16:35
  • 3
    @JosephR. you're absolutely right, my bad. I removed that suggestion. Thanks for pointing it out. – terdon Aug 1 '13 at 16:39
  • Hi there, thanks very much :) it's almost working, but there are two small things - it should be keeping the ones that match the other file, and the partial match for the perl doesn't seem to be quite working in a test I ran, it seems to be because all my lines don't have any spaces in them. I'm sorry for the noobishness, my perl skills aren't very good. – Joe Aug 1 '13 at 16:58
  • @Joe not a problem. Could you add some example input to your question so I can get a better idea of what you need? Also include your desired output. – terdon Aug 1 '13 at 17:00
  • Hi there, thanks very much, that's almost working for me, however I only seem to be getting full matches rather than partial matches. – Joe Aug 1 '13 at 17:23

If the files like in your example are sorted and always follow that pattern, you could write it:

join -t/ -1 3 -2 3 123.txt 789.txt |
  sed -n 's,\([^/]*/\)\([^/]*://\)\2,\2\1,p'

That would be the most efficient.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.