Sign up ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It's 100% free, no registration required.

I have a file containing two paths on each line. I want to remove the lines containing the same path twice.

I work on Linux and Solaris. I would like a one-liner in sed or awk or perl.

Example input file:

     /usr/lib/ /usr/lib/
     /usr/lib/ /usr/lib/
     /usr/lib/sse2/ /usr/lib/sse2/
     /usr/local/swp-tomcat-6.0/lib/commons-logging-1.1.1.jar /usr/local/swp-tomcat-6.0/lib/commons-logging-1.1.1.jar
     /usr/share/doc/libXrandr-1.1.1 /usr/share/doc/libXrandr-1.1.1
     /usr/share/doc/libxslt-1.1.17 /usr/share/doc/libxslt-1.1.17
     /etc/ /etc/
     /etc/ /etc/

Expected output:

     /etc/ /etc/
     /etc/ /etc/
share|improve this question
grep isn't good? That is the dedicated tool to output lines from a file matching certain condition: grep -vx '\s*\(\S\+\)\s\+\1\s*' file. – manatwork Jun 4 '13 at 10:47

3 Answers 3

up vote 4 down vote accepted
awk '{ if ($1 != $2 ) print $1" "$2; }' file

Just replace file for the appropriate file.

Or as @manatwork mentioned in the comments and simpler

awk '$1!=$2' file
share|improve this answer
Why not just awk '$1!=$2' file? – manatwork Jun 4 '13 at 10:40
I was not aware it could be that simple. :/ – BitsOfNix Jun 4 '13 at 10:43

You can express repeated text in grep's regexps (this is an extension to the mathematical notion of regular expression).

grep -v '^ *\([^ ][^ ]*\)  *\1 *$'

[^ ][^ ]* matches one or more non-space character. The backslash-parentheses make this a group, and \1 means “the same text as the first group”.

share|improve this answer
Except from being an alternative, can you give an argument why one should use grep over awk here? – Bernhard Jun 5 '13 at 8:35
@Bernhard Both are good methods. More people know grep basics than awk basics, but on the other hand the awk code is clearer, so I think there's no clear winner. – Gilles Jun 5 '13 at 8:55

This might work for you (GNU sed):

sed -r '/(\S+)\s\1/d' file
share|improve this answer
Though untested, your (elegant!) solution should also work in non-GNU sed when written like this: sed '/\(\S\+\)\s\1/d' file. Just a little more escaping needed, and you'll be 100% compatible everywhere. – syntaxerror Jun 19 at 11:18

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.