I have a text file containing lines like this:

This is a thread  139737522087680
This is a thread  139737513694976
This is a thread  139737505302272
This is a thread  139737312270080
.
.
.
This is a thread  139737203164928
This is a thread  139737194772224
This is a thread  139737186379520

How can I be sure of the uniqueness of every line?

NOTE: The goal is to test the file, not to modify it if duplicate lines are present.

NOTE 2: Submission for the moderator's information, the edit is okay.

up vote 24 down vote accepted
[ "$(wc -l < input)" -eq "$(sort -u input | wc -l)" ] && echo all unique
  • Exactly what I would have said, except with uniq instead of sort -u – Nonny Moose Jul 8 at 18:42
  • 1
    If the input is not already sorted, uniq would be a big mistake; it only deduplicates adjacent lines! – alexis Jul 9 at 13:26
  • If one is interested in the culprits, a sort <file> | uniq -d would print the duplicates. – Rolf Jul 10 at 8:08

Awk solution:

awk 'a[$0]++{print "dupes"; exit(1)}' file && echo "no dupes"
  • 4
    +1 The accepted answer reads through the whole file twice, while this stops as soon as it encounters a duplicate line in one read. This will also work with piped input, while the other needs files it can re-read. – JoL Jul 6 at 17:26
  • Couldn't you shove the echo into END? – Ignacio Vazquez-Abrams Jul 6 at 22:32
  • 2
    @IgnacioVazquez-Abrams There's really no point in the echo. Doing && echo or || echo is a convention in answers to indicate that a command does the right thing with the exit status code. The important thing is the exit(1). Ideally, you'd use this like if has_only_unique_lines file; then ..., not if [[ $(has_only_unique_lines file) = "no dupes" ]]; then ..., that'd be silly. – JoL Jul 7 at 1:10
  • 2
    Where other answers reads the file twice to save memory, this will read the whole file into memory, if there are no dupes. – Kusalananda Jul 8 at 17:29
  • 1
    @Kusalananda While this will read the whole file into memory when there are no dupes, using sort will too, regardless of whether there are dupes or not, right? How is that saving memory? – JoL Jul 9 at 0:50

Using sort/uniq:

sort input.txt | uniq

To check only for duplicate lines use the -d option for uniq. This will show only lines that are duplicate, if none it will show nothing:

sort input.txt | uniq -d
  • This is my goto. Not sure what the other, higher-voted answers offer that this one doesn't. – user1717828 Jul 6 at 20:00
  • 1
    It's good alternative to remove duplicates. – snr Jul 6 at 20:37
  • 1
    This doesn't do what he wants. He wants to know if there are duplicates, not remove them. – Barmar Jul 6 at 22:08
  • @Barmar: While it does seem that way the question is still unclear. As well as OPs comment attempting to clarify it. – Jesse_b Jul 6 at 22:09
  • There's a pending edit that adds more clarification. – Barmar Jul 6 at 22:24

TLDR

The original question was unclear, and read that the OP simply wanted a unique version of the contents of a file. That's shown below. In the since updated form of the question, the OP is now stating that he/she simply wants to know if the contents of the file is unique or not.


Test if file's contents is unique or not

You can simply use sort to verify if a file is unique or contains duplicates like so:

$ sort -uC input.txt && echo "unique" || echo "duplicates"

Example

Say I have these two files:

duplicate sample file
$ cat dup_input.txt
This is a thread  139737522087680
This is a thread  139737513694976
This is a thread  139737505302272
This is a thread  139737312270080
This is a thread  139737203164928
This is a thread  139737194772224
This is a thread  139737186379520
unique sample file
$  cat uniq_input.txt
A
B
C
D

Now when we analyze these files we can tell if they're unique or contain duplicates:

test duplicates file
$ sort -uC dup_input.txt && echo "unique" || echo "duplicates"
duplicates
test unique file
$ sort -uC uniq_input.txt && echo "unique" || echo "duplicates"
unique

Original question (unique contents of file)

Can be done with just sort:

$ sort -u input.txt
This is a thread  139737186379520
This is a thread  139737194772224
This is a thread  139737203164928
This is a thread  139737312270080
This is a thread  139737505302272
This is a thread  139737513694976
This is a thread  139737522087680

I usually sort the file, then use uniq to count the number of duplicates, then I sort once more see the duplicates at the bottom of the list.

I added one duplicate to the examples you provided:

$ sort thread.file | uniq -c | sort
      1 This is a thread  139737186379520
      1 This is a thread  139737194772224
      1 This is a thread  139737203164928
      1 This is a thread  139737312270080
      1 This is a thread  139737513694976
      1 This is a thread  139737522087680
      2 This is a thread  139737505302272

Since I haven't read the man page for uniq in awhile, I took a quick look for any alternatives. The following eliminates the need for the second sort, if you just want to see duplicates:

$ sort thread.file | uniq -d
This is a thread  139737505302272
  • It's a good alternative indeed. #rez – snr Jul 6 at 17:02

If there are no duplicates, all lines are unique:

[ "$(sort file | uniq -d)" ] && echo "some line(s) is(are) repeated"

Description: Sort the file lines to make repeated lines consecutive (sort)
Extract all consecutive lines that are equal (uniq -d).
If there is any output of the command above ([...]), then (&&) print a message.

This would not be complete without a Perl answer!

$ perl -ne 'print if ++$a{$_} == 2' yourfile

This will print each non-unique line once: so if it prints nothing, then the file has all unique lines.

Using cmp and sort in bash:

cmp -s <( sort file ) <( sort -u file ) && echo 'All lines are unique'

or

if cmp -s <( sort file ) <( sort -u file )
then
    echo 'All lines are unique'
else
    echo 'At least one line is duplicated'
fi

This would sort the file twice though, just like the accepted answer.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.