The uniq tag has no wiki summary.
10
votes
3answers
390 views
What is the difference between “sort -u” and “sort | uniq”?
Everywhere I see someone needing to get a sorted, unique list, they always pipe to sort | uniq. I've never seen any examples where someone uses sort -u instead. Why not? What's the difference, and why ...
0
votes
1answer
60 views
How to use uniq Unix command and specify input and output file?
I'm confusing use of uniq command, can anyone explain this command and how to use it?
How to use uniq without specifying any option?
Any example would be helped and appreciated
1
vote
1answer
126 views
How to match 2 large file and print the difference in shell script
I have 2 files.
File1 has 400k numerical records.
e.g:
1
2
3
4
5
6
.. and so on
File 2 also has 420k numerical records.
e.g:
1
2
3
4
6
.. and so on
Both these file are in unsorted manner. I want ...
4
votes
1answer
77 views
Why is uniq ignoring Unicode and lines with a single letter?
I'm trying to combine both the American and British dictionaries into one large dictionaries, and I'm trying to remove all the duplicates from the superset, but it seems that uniq is not outputting ...
1
vote
1answer
687 views
The simplest method to count lines matching specific patterns, including '0' if line is not found?
I have very big logs (several gigabytes per day), that can (but do not need to) contain specific lines. I have to count the number of occurences of every one of these lines on a daily basis.
I have a ...
0
votes
1answer
351 views
Unique lines based on the first field
How can you find unique lines or duplicate lines in a file based on the content of the first field, or the first n characters ?
The uniq command on OS X lacks the -w switch that is available under ...
6
votes
1answer
757 views
How can I remove duplicates in my .bash_history, preserving order?
I really enjoying using control+r to recursively search my command history. I've found a few good options I like to use with it:
# ignore duplicate commands, ignore commands starting with a space
...
4
votes
3answers
1k views
Get lines with maximum values in the column using awk, uniq and sort
I have a file with next format
2011-12-01 user1 access1
2011-12-01 user1 access2
2011-12-01 user2 access2
2011-12-01 user4 access2
2011-12-02 user1 access1
2012-01-01 user3 access1
2012-01-01 user4 ...
10
votes
5answers
2k views
case-insensitive search of duplicate file-names
I there a way to find all files in a directory with duplicate filenames, regardless of the casing (upper-case and/or lower-case)?
1
vote
3answers
483 views
How let `sort | uniq -c` separate the number of occurrences by a tab charactor?
uniq -c separate the occurrence number by spaces, which is hard for to cut or awk to separate it out later.
1000_A1\tB1\n
___1_A2\tB2\n
I can solve this problem using sed -r 's/^ *([0-9]+)/\1\t/' ...
7
votes
1answer
1k views
How to remove duplicate lines in a large multi-GB textfile?
My question is similar to this question but with a couple of different constraints:
I have a large \n delimited wordlist -- one word per line. Size of
files range from 2GB to as large as 10GB.
I ...
3
votes
2answers
547 views
Where has my `uniq` or `sort -u` line gone, with some unicode characters
What is going on in the following code snippet? I'm not getting my expected output.
I'd think it was a bug, but it happens for 2 different programs (uniq and sort), so I suspect it is something ...
10
votes
2answers
896 views
How to get only the unique results without having to sort data?
$ cat data.txt
aaaaaa
aaaaaa
cccccc
aaaaaa
aaaaaa
bbbbbb
$ cat data.txt | uniq
aaaaaa
cccccc
aaaaaa
bbbbbb
$ cat data.txt | sort | uniq
aaaaaa
bbbbbb
cccccc
$
The result that I need is to display ...
