Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Help me to print unique lines in Linux. Word examples like

A B
B C
C D
A E
F G
H I
I J

the output should be

F G
H I

I.e., the line should be unique at first and the words in the line should also appear only once as in above example “F G” is a unique line and neither “F” or “G” never exists in any other line similarly “H I”.

share|improve this question
2  
Well, "I" appears twice. Do you mean then should not appear in previous lines? –  Stéphane Chazelas Nov 5 '12 at 11:07

2 Answers 2

You could get the list of unique words. And then, matching lines would be lines whose all words are unique:

tr -cs 'A-Z' '[\n*]' < words.txt |
  sort |
  uniq -u |
  perl -lne '
    if ($ARGV eq "-") {
      $u{$_}=1;
      next
    }
    for $w (/[A-Z]+/g) {
      next LINE unless $u{$w}
    }
    print' - words.txt

Change A-Z to the list of characters that constitute a word.

share|improve this answer

You could do it like this with awk:

unique.awk

FNR == NR {
  for(i=1; i<=NF; i++)
    if(++w[$i] > 1)
      not_unique[$i] = 1
  next
}

{
  for(i=1; i<=NF; i++)
    if(not_unique[$i])
      next
}

1

Run it like this:

awk -f unique.awk infile infile

Output:

F G

As a one-liner:

awk 'FNR == NR { for(i=1; i<=NF; i++) if(++w[$i] > 1) not_unique[$i] = 1; next } { for(i=1; i<=NF; i++) if(not_unique[$i]) next } 1' infile infile

Explanation

The file needs to be parsed twice, first to find all non-unique words, then to print those lines which contain only unique words. This is reflected in the program structure, the first block creates a hash containing words that are not unique, the second checks each line and skips it if it has non-unique words. The trailing 1 at the end is only reached when unique lines occur and it invokes awk's default action ({ print $0 }).

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.