The tag has no wiki summary.

learn more… | top users | synonyms

17
votes
3answers
4k views

What is the difference between “sort -u” and “sort | uniq”?

Everywhere I see someone needing to get a sorted, unique list, they always pipe to sort | uniq. I've never seen any examples where someone uses sort -u instead. Why not? What's the difference, and why ...
16
votes
4answers
3k views

How to get only the unique results without having to sort data?

$ cat data.txt aaaaaa aaaaaa cccccc aaaaaa aaaaaa bbbbbb $ cat data.txt | uniq aaaaaa cccccc aaaaaa bbbbbb $ cat data.txt | sort | uniq aaaaaa bbbbbb cccccc $ The result that I need is to display ...
14
votes
2answers
617 views

What did `uniq -t` do?

I have some old code from 2003 which uses -t option for uniq command. It throws an error since that option is probably not supported anymore. Here's the piece which uses the command: egrep -n "{ ...
10
votes
5answers
3k views

case-insensitive search of duplicate file-names

I there a way to find all files in a directory with duplicate filenames, regardless of the casing (upper-case and/or lower-case)?
8
votes
3answers
2k views

How can I remove duplicates in my .bash_history, preserving order?

I really enjoying using control+r to recursively search my command history. I've found a few good options I like to use with it: # ignore duplicate commands, ignore commands starting with a space ...
7
votes
1answer
2k views

How to remove duplicate lines in a large multi-GB textfile?

My question is similar to this question but with a couple of different constraints: I have a large \n delimited wordlist -- one word per line. Size of files range from 2GB to as large as 10GB. I ...
4
votes
3answers
3k views

Get lines with maximum values in the column using awk, uniq and sort

I have a file with next format 2011-12-01 user1 access1 2011-12-01 user1 access2 2011-12-01 user2 access2 2011-12-01 user4 access2 2011-12-02 user1 access1 2012-01-01 user3 access1 2012-01-01 user4 ...
4
votes
1answer
154 views

Why is uniq ignoring Unicode and lines with a single letter?

I'm trying to combine both the American and British dictionaries into one large dictionaries, and I'm trying to remove all the duplicates from the superset, but it seems that uniq is not outputting ...
3
votes
2answers
880 views

Where has my `uniq` or `sort -u` line gone, with some unicode characters

What is going on in the following code snippet? I'm not getting my expected output. I'd think it was a bug, but it happens for 2 different programs (uniq and sort), so I suspect it is something ...
2
votes
4answers
68 views

print parent directories full path of find output

what I want to do is to find all files based on some search query and get parents parent directory (../..) full path. For example find . -name "__openerp__.py" and then for each file execute ...
2
votes
1answer
2k views

The simplest method to count lines matching specific patterns, including '0' if line is not found?

I have very big logs (several gigabytes per day), that can (but do not need to) contain specific lines. I have to count the number of occurences of every one of these lines on a daily basis. I have a ...
2
votes
3answers
56 views

I need data from a csv file which is not present in the second csv file

For example: I have a CSV file (first file ) with below data (vertically): 1 2 3 4 5 6 Second CSV with below data (vertically): 5 6 7 8 I want my resultant file (action needs to be performed ...
2
votes
1answer
61 views

Sort doesn't use natural order

I have used uniq -c and sort extensively on other machines, and it always worked very well. Now, on my new Ubuntu (13.04) machine, I end up with a weird ordering: % cat queries.log | grep INSERT | ...
2
votes
3answers
36 views

How can I get the last line number of unique commands when grepping my history file?

I rely on my history file for previous commands, especially ones that are long with many parameters. Once I find the command and its line number(nn) I use !nn to run it. I prefer to see the full ...
2
votes
2answers
35 views

Check if a drive has been Zeroed with xxd and uniq?

I'm looking for a way to check if the entire binary contents of a USB device are purely 0s. I'm trying this command: sudo xxd /dev/sdb | uniq but it has no output and just runs forever. I've ...
2
votes
3answers
70 views

get list of months using dseq from dateutils

how'd I get a listing of the months in the year (abbreviated) using dseq from dateutils? NOTE simply running: dseq 'jan' 'dec' -i '%b' -f '%b' returns a humongous list of months: piping the ...
1
vote
1answer
581 views

How to match 2 large file and print the difference in shell script

I have 2 files. File1 has 400k numerical records. e.g: 1 2 3 4 5 6 .. and so on File 2 also has 420k numerical records. e.g: 1 2 3 4 6 .. and so on Both these file are in unsorted manner. I want ...
1
vote
3answers
876 views

How let `sort | uniq -c` separate the number of occurrences by a tab charactor?

uniq -c separate the occurrence number by spaces, which is hard for to cut or awk to separate it out later. 1000_A1\tB1\n ___1_A2\tB2\n I can solve this problem using sed -r 's/^ *([0-9]+)/\1\t/' ...
0
votes
1answer
1k views

Unique lines based on the first field

How can you find unique lines or duplicate lines in a file based on the content of the first field, or the first n characters ? The uniq command on OS X lacks the -w switch that is available under ...
0
votes
1answer
37 views

Uniq won't remove duplicate

I was using the following command curl -silent http://api.openstreetmap.org/api/0.6/relation/2919627 http://api.openstreetmap.org/api/0.6/relation/2919628 | grep node | awk '{print $3}' | uniq ...
0
votes
1answer
368 views

How to use uniq Unix command and specify input and output file?

I'm confusing use of uniq command, can anyone explain this command and how to use it? How to use uniq without specifying any option? Any example would be helped and appreciated
0
votes
1answer
61 views

uniq and sed, delete lines with pattern similar in multiple files

I need delete lines with pattern similar, in multiple files (2500 .in files). example I have an input file -- "transmission-gtk.in" -- whose content is something like, but it is variable, I do not ...