0

I have two text files containing a list of URLs. I want to compare the two files and print out the differences. I know that we can use 'diff' but here's the problem. The urls on one file are full e.g. http://google.com while on the other file they don't have the http://, e.g. google.com

File 1:

http://google.com

File 2:

google.com

I want these to be ignored and only actually different URLs to be printed. Is it possible?

  • 1
    Can you show us a slightly longer sample of both files, or at least a longer list of examples? – EightBitTony Jan 21 '16 at 13:02
  • Yes , you can write a python script to do that. – Ijaz Ahmad Khan Jan 21 '16 at 13:03
  • 2
    Still not enough - do the two files have a sorted list of URL's or are they a random assortment of URLs that may not be in the same order in each file? Are the full URLs or just domain names prefixed by http:// (i.e. do the URLs contain slashes, punctuation, numbers, etc.) Does the order matter, or is it purely 'does this URL exist in this other file'. We need to see a real sample set or a more detailed description and as @don_crissti suggests, we need to know what output you're expecting. – EightBitTony Jan 21 '16 at 13:24
  • Isn't this what diff does? – DisplayName Jan 21 '16 at 17:26
  • Unfortunately, i don't have a longer list of examples. The list are not in the same order, so we would need to sort it first. Some of the URLs do contain slashes and punctuation. The order doesn't matter, i just need to check whether a URL exists in the other file. Thanks! – Koshur Jan 24 '16 at 7:02
0

Take, for example, these two files:

$ cat file1
http://google.com
example.com
http://foobar.org

$ cat file2
example.com
google.com
foobar.org
unique.url

I would just use a tool like sed to remove everything up to the last occurrence of //. Also, in order to use diff, both files need to be sorted. Combining the two gives us:

$ diff <(sed 's#.*//##' file1 | sort) <(sort file2) 
3a4
> unique.url

Alternatively, use comm -3:

$ comm -3 <(sed 's#.*//##' file1 | sort) <(sort file2) 
    unique.url

You can remove the leading whitespace with sed again:

$ comm -3 <(sed 's#.*//##' file1 | sort) <(sort file2) | sed 's/^\s*//'
unique.url

Another approach would be to remove the http:// from both files and print what remains, then pass it through uniq -u which will only print unique lines. Since those URLs which are present in both files won't be unique, this will only print the ones that are present on one of the two files:

$ sed 's#.*//##' file1 file2 | sort | uniq -u
unique.url
| improve this answer | |
  • Is it possible to check each line against all the lines in other file? I mean what if even after sorting, some hosts exist in different lines? – Koshur Jan 25 '16 at 12:12
  • @Koshur I don't understand. This does check everything. By the way, if one of the answers here solved your issue, please take a moment and accept it by clicking on the check mark to the left. That will mark the question as answered and is the way thanks are expressed on the Stack Exchange sites. – terdon Jan 25 '16 at 12:14
3

Yes, you can e.g. use sed to remove certain things before piping it into diff:

$ diff file1 file2
1,3c1,3
< http://google.com/search
< http://www.google.com
< http://example.com
---
> google.com/search
> google.com
> example.com

$ < file1 sed 's|https\{0,1\}://||g' | diff - file2
2c2
< www.google.com
---
> google.com
| improve this answer | |
  • @StéphaneChazelas please explain your edit. I'd like to understand this < file1 sed magic. – s3lph Jan 21 '16 at 16:24
  • 1
    That's just redirection. Redirections can be anywhere on the line. cmd arg < file, cmd < file arg, < file cmd arg. Some people do UUOCs because they like to have the input on the left. If you're more comfortable with the redirection on the right, you can change it to sed 's|https\{0,1\}://||g' < file | diff - file2` or since sed can also open the file by itself: sed 's|https\{0,1\}://||g' file1 | diff - file2. – Stéphane Chazelas Jan 21 '16 at 16:33
0

Yes, you can skip the part (e.g. http://) with awk's field separator.

from manpage:

NAME
       awk - pattern scanning and processing language

-F fs
       --field-separator fs
              Use fs for the input field separator (the value of the FS predefined variable).

Example:

$ cat file1
http://google.com
http://gnu.org
http://fsf.org
http://linux.stackexchange.com

$ cat file2
google.com
gnu.org
fsf.org
unix.stackexchange.com

$ cat file1 | awk -F "http://" '{print $2}'
google.com
gnu.org
fsf.org
linux.stackexchange.com

$ cat file1 | awk -F "http://" '{print $2}' | diff - file2
4c4
< linux.stackexchange.com
---
> unix.stackexchange.com

Note:

  • You can also use -F "://" generally for https:// or http://
  • The files should be sorted before comparision
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.