Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have two files, let's call them 123.txt and 789.txt. 123.txt is 2.5M lines long, and 789.txt is 65M lines long. Is there any way to use grep or similar to keep any lines from 789.txt that contain lines from 123.txt?

There will be a max of one duplicate per line in 789.txt, and the duplicate text will be at the beginning of the line. I'm totally stuck on this, and couldn't find any info online, so I don't really have anything to start with. It will be running on a server, so I don't mind it taking a while (which I know it will)

I have two files, let's call them 123.txt and 789.txt. 123.txt is 2.5M lines long, and 789.txt is 65M lines long. Is there any way to use grep or similar to remove any lines from 789.txt that contain lines from 123.txt?

There will be a max of one duplicate per line in 789.txt, and the duplicate text will be at the beginning of the line. I'm totally stuck on this, and couldn't find any info online, so I don't really have anything to start with. It will be running on a server, so I don't mind it taking a while (which I know it will)

  • 123.txt:

    hxxp://www.a.com
    hxxp://www.b.com
    hxxp://www.c.com
    
  • 789.txt:

    hxxp://www.a.com/kgjdk-jgjg/
    hxxp://www.b.com/gsjahk123/
    hxxp://www.c.com/abc.txt
    hxxp://www.d.com/sahgsj/
    
  • Desired output:

    hxxp://www.a.com/kgjdk-jgjg/
    hxxp://www.b.com/gsjahk123/
    hxxp://www.c.com/abc.txt
    
share|improve this question

2 Answers 2

up vote 7 down vote accepted

You can do this very easily using grep:

$ grep -Ff 123.txt 789.txt
http://www.a.com/kgjdk-jgjg/ 
http://www.b.com/gsjahk123/ 
http://www.c.com/abc.txt 

The command above, will print all lines from file 789.txt that contain any of the lines from 123.txt. The -f means "read the patterns to search from this file" and the -F tells grep to treat the search patterns as strings and not its default regular expressions.

This will not work if the lines of 123.txt contain trailing spaces, grep will treat the spaces as part of the pattern to look for an will not match if it occurs within a word. For example, the pattern foo (note the trailing space) will not match foobar. To remove trailing spaces from your file, run this command:

$ sed 's/ *$//' 123.txt > new_file

Then use the new_file to grep:

$ grep -Ff 123.txt 789.txt

You can also do this without a new file, using the i flag:

$ sed -i.bak 's/ *$//' 123.txt

This will change file 123.txt and keep a copy of the original called 123.txt.bak.

share|improve this answer
    
I don't think the first option would work, regardless of the OP's case: it would print lines in 123.txt that aren't in 789.txt as well as lines in 789.txt that appear in 123.txt (they will only be printed once, but printed nonetheless). –  Joseph R. Aug 1 '13 at 16:35
2  
@JosephR. you're absolutely right, my bad. I removed that suggestion. Thanks for pointing it out. –  terdon Aug 1 '13 at 16:39
    
Hi there, thanks very much :) it's almost working, but there are two small things - it should be keeping the ones that match the other file, and the partial match for the perl doesn't seem to be quite working in a test I ran, it seems to be because all my lines don't have any spaces in them. I'm sorry for the noobishness, my perl skills aren't very good. –  Joe Aug 1 '13 at 16:58
    
@Joe not a problem. Could you add some example input to your question so I can get a better idea of what you need? Also include your desired output. –  terdon Aug 1 '13 at 17:00
    
Hi there, thanks very much, that's almost working for me, however I only seem to be getting full matches rather than partial matches. –  Joe Aug 1 '13 at 17:23

If the files like in your example are sorted and always follow that pattern, you could write it:

join -t/ -1 3 -2 3 123.txt 789.txt |
  sed -n 's,\([^/]*/\)\([^/]*://\)\2,\2\1,p'

That would be the most efficient.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.