3
$ grep home american.txt | sort | uniq                                                                                            
A home and a country should leave us no more!                                                                                        
Between their loved home and the war's desolation!                                                                                 
O'er the land of the free and the home of the brave!                                                                                  
O'er the land of the free and the home of the brave!                                                                                  
O'er the land of the free and the home of the brave?     

Why is this showing two duplicate lines?

Here's the output of grep home american.txt | cat -A:

O'er the land of the free and the home of the brave?^M$
O'er the land of the free and the home of the brave!^M$
A home and a country should leave us no more! ^M$
O'er the land of the free and the home of the brave!^M$
Between their loved home and the war's desolation!^M$
O'er the land of the free and the home of the brave!$
  • 3
    It can be different numbers of white spaces at the end of the line. – David Dai Oct 9 '15 at 3:55
  • I checked, there were no spaces at the end of any of those lines. – WinMacLinUser Oct 9 '15 at 3:58
  • Slightly unrelated, but how did this end up here:pastebin.com/UrGSg9qg – WinMacLinUser Oct 9 '15 at 4:01
  • 1
    please upload to somewhere that allows download rather than cut-and-paste. when i select all that text it's obvious that there is white-space at the end of each line. btw, if you have GNU cat, try piping the ouput from uniq into cat -A. – cas Oct 9 '15 at 4:13
  • 4
    one of those lines ending in brave! has a carriage-return (^M), the other doesn't. uniq is right, they're different. – cas Oct 9 '15 at 4:18
1

O'er the land of the free and the home of the brave!

There are 3 occurrences of the line above. The first two occurrences both have a carriage return at the end. The 3rd occurrence doesn't. you can use the following command to visualize it:

cat -A american.txt

It's caused by the fact that this text file lacks an empty line at the end.
I think it's a convention in the Unix/Linux world to always have an empty line at the end of text files.

Some other things I found interesting while I was testing it:

  • the command works differently in cygwin.
  • grep home american.txt | sort| uniq -u works just fine.
  • i'm more amused than annoyed, but that's just rude. i found the answer and explained it in the comments and you posted my work as your answer. – cas Oct 9 '15 at 4:40
  • @cas I'm so so sorry. I was thinking straight. – David Dai Oct 9 '15 at 4:42
  • that's ok, it's a trivial Q, and a trivial A which is why i didn't bother posting it as an Answer myself. – cas Oct 9 '15 at 4:43
  • @cas what do you mean 'trivial' – WinMacLinUser Oct 9 '15 at 4:46
  • i mean not difficult. – cas Oct 9 '15 at 4:47
2

As suspected by David Dai and cas the two lines are in fact different, but the only difference is in invisible characters.

Your file is a Windows text file. In a Windows text files, lines are separated by the two-character sequence CR, LF (carriage return, line feed). In a Unix text file, lines are terminated by a LF (line feed, also known as newline) character. cat -A shows a CR as ^M and a line feed as $ followed by a line break.

When a Windows text file is processed by a Unix utility, the Unix utility sees an extra CR character at the end of every line. As far as Unix utilities are concerned, CR is just an ordinary character; a line with a CR at the end is different from a line that's identical except for the lack of a CR at the end. Furthermore, on Windows, the newline sequence is a separator, so there is no CR, LF at the end of the file. But on Unix, a text file always ends with a LF character unless it's empty. So when you process a Windows text file with a Unix utility, what the Unix utility sees is a file with CR at the end of every line (because CR characters aren't part of the Unix encoding of a newline), and some trailing text that isn't part of a line (because of the lack of a newline at the end).

Unix text utilities differ in what they do when their input is not a valid text file due to the lack of a final newline. GNU utilities — which is what you'll find on non-embedded Linux and on Cygwin — strive to treat such files as text file and preserve the lack of a final newline. The sort command shuffles lines around, and while it does process an unfinished input line, it always emits a newline at the end of the output. Thus to the sort command what you have looks like a bunch of lines, each of which ends with the character CR, except the last input line which doesn't end with CR. In the output, all the lines end with CR except that one line that corresponds to the last input line.

uniq sees a bunch of lines that end with brave! and a CR, and retains only one of them. It also sees one line that ends with brave! but no CR, which it dutifully emits since that line differs from every other line.

When you print out the output on the terminal, a CR character instructs the terminal to move the cursor to the beginning of the current line; a LF character instructs the terminal to move the cursor to the beginning of the next line. Thus the sequences LF and CR,LF are not visually distinguishable, and you see two identical-looking lines. The command cat -A adds printable characters to make them distinct.

1

One of those lines ending in brave! has a carriage-return (^M), the other doesn't. uniq is right, they're different.

All lines in the file except the very last one have carriage returns. It has them because it was created by an MS-DOS or Windows text-editor (which uses CR/LF for line-endings instead of the unixish LF-only or \n). The last line in the file doesn't have CR or LF (otherwise there'd be a blank line beneath it).

That's why, for example, if you tail -1 american.txt you get the next shell prompt on the same line as the text, without a line-feed break.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.