Check all lines of a file are unique

Question

I have a text file containing lines like this:

This is a thread  139737522087680
This is a thread  139737513694976
This is a thread  139737505302272
This is a thread  139737312270080
.
.
.
This is a thread  139737203164928
This is a thread  139737194772224
This is a thread  139737186379520

How can I be sure of the uniqueness of every line?

NOTE: The goal is to test the file, not to modify it if duplicate lines are present.

Answer 1

[ "$(wc -l < input)" -eq "$(sort -u input | wc -l)" ] && echo all unique

Answer 2

Using sort/uniq:

sort input.txt | uniq

To check only for duplicate lines use the -d option for uniq. This will show only lines that are duplicate, if none it will show nothing:

sort input.txt | uniq -d

Answer 3

Awk solution:

awk 'a[$0]++{print "dupes"; exit(1)}' file && echo "no dupes"

Answer 4

TLDR

The original question was unclear, and read that the OP simply wanted a unique version of the contents of a file. That's shown below. In the since updated form of the question, the OP is now stating that he/she simply wants to know if the contents of the file is unique or not.

---

Test if file's contents is unique or not

You can simply use sort to verify if a file is unique or contains duplicates like so:

$ sort -uC input.txt && echo "unique" || echo "duplicates"

Example

Say I have these two files:

duplicate sample file

$ cat dup_input.txt
This is a thread  139737522087680
This is a thread  139737513694976
This is a thread  139737505302272
This is a thread  139737312270080
This is a thread  139737203164928
This is a thread  139737194772224
This is a thread  139737186379520

unique sample file

$  cat uniq_input.txt
A
B
C
D

Now when we analyze these files we can tell if they're unique or contain duplicates:

test duplicates file

$ sort -uC dup_input.txt && echo "unique" || echo "duplicates"
duplicates

test unique file

$ sort -uC uniq_input.txt && echo "unique" || echo "duplicates"
unique

Original question (unique contents of file)

Can be done with just sort:

$ sort -u input.txt
This is a thread  139737186379520
This is a thread  139737194772224
This is a thread  139737203164928
This is a thread  139737312270080
This is a thread  139737505302272
This is a thread  139737513694976
This is a thread  139737522087680

Answer 5

I usually sort the file, then use uniq to count the number of duplicates, then I sort once more see the duplicates at the bottom of the list.

I added one duplicate to the examples you provided:

$ sort thread.file | uniq -c | sort
      1 This is a thread  139737186379520
      1 This is a thread  139737194772224
      1 This is a thread  139737203164928
      1 This is a thread  139737312270080
      1 This is a thread  139737513694976
      1 This is a thread  139737522087680
      2 This is a thread  139737505302272

Since I haven't read the man page for uniq in awhile, I took a quick look for any alternatives. The following eliminates the need for the second sort, if you just want to see duplicates:

$ sort thread.file | uniq -d
This is a thread  139737505302272

Answer 6

If there are no duplicates, all lines are unique:

[ "$(sort file | uniq -d)" ] && echo "some line(s) is(are) repeated"

Description: Sort the file lines to make repeated lines consecutive (sort)
Extract all consecutive lines that are equal (uniq -d).
If there is any output of the command above ([...]), then (&&) print a message.

Answer 7

This would not be complete without a Perl answer!

$ perl -ne 'print if ++$a{$_} == 2' yourfile

This will print each non-unique line once: so if it prints nothing, then the file has all unique lines.

Answer 8

Using cmp and sort in bash:

cmp -s <( sort file ) <( sort -u file ) && echo 'All lines are unique'

or

if cmp -s <( sort file ) <( sort -u file )
then
    echo 'All lines are unique'
else
    echo 'At least one line is duplicated'
fi

This would sort the file twice though, just like the accepted answer.