I have a file containing two paths on each line. I want to remove the lines containing the same path twice.

I work on Linux and Solaris. I would like a one-liner in sed or awk or perl.

Example input file:

     /usr/lib/libgmp.so.3.3.3 /usr/lib/libgmp.so.3.3.3
     /usr/lib/libxslt.so.1.1.17 /usr/lib/libxslt.so.1.1.17
     /usr/lib/sse2/libgmp.so.3.3.3 /usr/lib/sse2/libgmp.so.3.3.3
     /usr/local/swp-tomcat-6.0/lib/commons-logging-1.1.1.jar /usr/local/swp-tomcat-6.0/lib/commons-logging-1.1.1.jar
     /usr/share/doc/libXrandr-1.1.1 /usr/share/doc/libXrandr-1.1.1
     /usr/share/doc/libxslt-1.1.17 /usr/share/doc/libxslt-1.1.17
     /etc/ /etc/
     /etc/ /etc/

Expected output:

     /etc/ /etc/
     /etc/ /etc/
  • grep isn't good? That is the dedicated tool to output lines from a file matching certain condition: grep -vx '\s*\(\S\+\)\s\+\1\s*' file. – manatwork Jun 4 '13 at 10:47
awk '{ if ($1 != $2 ) print $1" "$2; }' file

Just replace file for the appropriate file.

Or as @manatwork mentioned in the comments and simpler

awk '$1!=$2' file
  • 5
    Why not just awk '$1!=$2' file? – manatwork Jun 4 '13 at 10:40
  • I was not aware it could be that simple. :/ – BitsOfNix Jun 4 '13 at 10:43

You can express repeated text in grep's regexps (this is an extension to the mathematical notion of regular expression).

grep -v '^ *\([^ ][^ ]*\)  *\1 *$'

[^ ][^ ]* matches one or more non-space character. The backslash-parentheses make this a group, and \1 means “the same text as the first group”.

  • 1
    Except from being an alternative, can you give an argument why one should use grep over awk here? – Bernhard Jun 5 '13 at 8:35
  • 2
    @Bernhard Both are good methods. More people know grep basics than awk basics, but on the other hand the awk code is clearer, so I think there's no clear winner. – Gilles 'SO- stop being evil' Jun 5 '13 at 8:55

This might work for you (GNU sed):

sed -r '/(\S+)\s\1/d' file
  • Though untested, your (elegant!) solution should also work in non-GNU sed when written like this: sed '/\(\S\+\)\s\1/d' file. Just a little more escaping needed, and you'll be 100% compatible everywhere. – syntaxerror Jun 19 '15 at 11:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.