# uniq showing duplicate lines

``````\$ grep home american.txt | sort | uniq
A home and a country should leave us no more!
Between their loved home and the war's desolation!
O'er the land of the free and the home of the brave!
O'er the land of the free and the home of the brave!
O'er the land of the free and the home of the brave?
``````

Why is this showing two duplicate lines?

Here's the output of `grep home american.txt | cat -A`:

``````O'er the land of the free and the home of the brave?^M\$
O'er the land of the free and the home of the brave!^M\$
A home and a country should leave us no more! ^M\$
O'er the land of the free and the home of the brave!^M\$
Between their loved home and the war's desolation!^M\$
O'er the land of the free and the home of the brave!\$
``````
• It can be different numbers of white spaces at the end of the line. Commented Oct 9, 2015 at 3:55
• I checked, there were no spaces at the end of any of those lines. Commented Oct 9, 2015 at 3:58
• Slightly unrelated, but how did this end up here:pastebin.com/UrGSg9qg Commented Oct 9, 2015 at 4:01
• please upload to somewhere that allows download rather than cut-and-paste. when i select all that text it's obvious that there is white-space at the end of each line. btw, if you have GNU cat, try piping the ouput from uniq into `cat -A`.
– cas
Commented Oct 9, 2015 at 4:13
• one of those lines ending in `brave!` has a carriage-return (^M), the other doesn't. uniq is right, they're different.
– cas
Commented Oct 9, 2015 at 4:18

O'er the land of the free and the home of the brave!

There are 3 occurrences of the line above. The first two occurrences both have a carriage return at the end. The 3rd occurrence doesn't. you can use the following command to visualize it:

cat -A american.txt

It's caused by the fact that this text file lacks an empty line at the end.
I think it's a convention in the Unix/Linux world to always have an empty line at the end of text files.

Some other things I found interesting while I was testing it:

• the command works differently in cygwin.
• `grep home american.txt | sort| uniq -u` works just fine.
• i'm more amused than annoyed, but that's just rude. i found the answer and explained it in the comments and you posted my work as your answer.
– cas
Commented Oct 9, 2015 at 4:40
• @cas I'm so so sorry. I was thinking straight. Commented Oct 9, 2015 at 4:42
• that's ok, it's a trivial Q, and a trivial A which is why i didn't bother posting it as an Answer myself.
– cas
Commented Oct 9, 2015 at 4:43
• @cas what do you mean 'trivial' Commented Oct 9, 2015 at 4:46
• By the way, @cas, please do not post answers in comments. If you mean to answer the question, post an answer. If you post something in a comment, it is absolutely not rude for someone else to post the same idea in an answer — you effectively gave up your turn. Commented Oct 10, 2015 at 0:01

As suspected by David Dai and cas the two lines are in fact different, but the only difference is in invisible characters.

Your file is a Windows text file. In a Windows text files, lines are separated by the two-character sequence CR, LF (carriage return, line feed). In a Unix text file, lines are terminated by a LF (line feed, also known as newline) character. `cat -A` shows a CR as `^M` and a line feed as `\$` followed by a line break.

When a Windows text file is processed by a Unix utility, the Unix utility sees an extra CR character at the end of every line. As far as Unix utilities are concerned, CR is just an ordinary character; a line with a CR at the end is different from a line that's identical except for the lack of a CR at the end. Furthermore, on Windows, the newline sequence is a separator, so there is no CR, LF at the end of the file. But on Unix, a text file always ends with a LF character unless it's empty. So when you process a Windows text file with a Unix utility, what the Unix utility sees is a file with CR at the end of every line (because CR characters aren't part of the Unix encoding of a newline), and some trailing text that isn't part of a line (because of the lack of a newline at the end).

Unix text utilities differ in what they do when their input is not a valid text file due to the lack of a final newline. GNU utilities — which is what you'll find on non-embedded Linux and on Cygwin — strive to treat such files as text file and preserve the lack of a final newline. The `sort` command shuffles lines around, and while it does process an unfinished input line, it always emits a newline at the end of the output. Thus to the `sort` command what you have looks like a bunch of lines, each of which ends with the character CR, except the last input line which doesn't end with CR. In the output, all the lines end with CR except that one line that corresponds to the last input line.

`uniq` sees a bunch of lines that end with `brave!` and a CR, and retains only one of them. It also sees one line that ends with `brave!` but no CR, which it dutifully emits since that line differs from every other line.

When you print out the output on the terminal, a CR character instructs the terminal to move the cursor to the beginning of the current line; a LF character instructs the terminal to move the cursor to the beginning of the next line. Thus the sequences LF and CR,LF are not visually distinguishable, and you see two identical-looking lines. The command `cat -A` adds printable characters to make them distinct.

One of those lines ending in `brave!` has a carriage-return (`^M`), the other doesn't. `uniq` is right, they're different.

All lines in the file except the very last one have carriage returns. It has them because it was created by an MS-DOS or Windows text-editor (which uses CR/LF for line-endings instead of the unixish LF-only or \n). The last line in the file doesn't have CR or LF (otherwise there'd be a blank line beneath it).

That's why, for example, if you `tail -1 american.txt` you get the next shell prompt on the same line as the text, without a line-feed break.