Tagged Questions

The tag has no wiki summary.

learn more… | top users | synonyms

1
vote
1answer
39 views

Removing duplicate lines after comparing first pattern and special pattern

I'm looking for code to removing duplicate lines after comparing the first tab-delimited pattern: Example : (↦ = tab) car ↦ ab ↦ 38/43 car ↦ ca ↦ 2/34 house ↦ ca ↦ 34/34 house ↦ d ↦ 2/3 house ↦ ht ↦ ...
1
vote
5answers
82 views

How to create an array of unique elements from a string/array in bash?

If I have a string "1 2 3 2 1" - or an array [1,2,3,2,1] - how can I select the unique values, i.e. "1 2 3 2 1" produces "1 2 3" or [1,2,3,2,1] produces [1,2,3] Similar to uniq but uniq seems ...
0
votes
0answers
18 views

How to find number of unique 'applications' in Debian jessie

This is a follow-up query on allpackages.txt.gz has two different file-sizes? $ wc -l allpackages.txt 57725 allpackages.txt I looked at the output and it says 57725 -6 the header files which is ...
12
votes
2answers
555 views

How does awk '!a[$0]++' work?

This one-liner removes duplicate lines from text input without pre-sorting. For example: $ cat >f q w e w r $ awk '!a[$0]++' <f q w e r $ The original code I have found on the internets ...
1
vote
2answers
62 views

How to match a number in one file to a name in another file?

Okay, I hope this is the right place to ask. I have 2 files. The first file has 4 fields: class CRN #, Instructor ID #, Class Section #, Number of Students. Example: 0002 T00005 006 15 0004 ...
1
vote
2answers
43 views

Getting the size of parent folders when using find to search for criteria?

I am hunting for a certain regular expression within the filenames and am seeking to return the parent folders with their sizes in a unique fashion. For example, here is the command that I expected ...
2
votes
1answer
36 views

Why might these values not be uniq'ed together?

I am trying to figure out which IP addresses hit the server most often. I received this surprising output from uniq: $ cat access_log | awk '{print $1}' | uniq -c | sort -bnr | head 26 73.23.5.25 ...
2
votes
2answers
93 views

Removing lines with a single common field

I have a tab separated file that looks like this: 123 some text 123 some different text 334 some other text 341 more text and I want to do two things. One is to order everything numerically ...
7
votes
4answers
207 views

Groupwise uniq command?

I am searching for a command to get from a file in this format: hello 32 hello 67 hi 2 ho 1212 ho 1390 ho 3000 To this format (deduplicate by taking the last row of a "group"): hello ...
3
votes
1answer
291 views

Fastest `uniq` tool in linux

I have large text file (1.5 G), I want to know what the fastest and more reliable tool in linux is. I usualy use: awk '!x[$0]++' file.txt But when I use htop command I see my memory usage is ...
2
votes
2answers
89 views

A combination of uniq and agrep?

I have a file full of long SQL queries, one per line. I need to create a list of unique queries, but most of the queries include parameter values that make using an exact matching tool like uniq ...
2
votes
2answers
105 views

Check if a drive has been Zeroed with xxd and uniq?

I'm looking for a way to check if the entire binary contents of a USB device are purely 0s. I'm trying this command: sudo xxd /dev/sdb | uniq but it has no output and just runs forever. I've ...
2
votes
3answers
365 views

How can I get the last line number of unique commands when grepping my history file?

I rely on my history file for previous commands, especially ones that are long with many parameters. Once I find the command and its line number(nn) I use !nn to run it. I prefer to see the full ...
0
votes
1answer
115 views

uniq and sed, delete lines with pattern similar in multiple files

I need delete lines with pattern similar, in multiple files (2500 .in files). example I have an input file -- "transmission-gtk.in" -- whose content is something like, but it is variable, I do not ...
0
votes
1answer
664 views

Uniq won't remove duplicate

I was using the following command curl -silent http://api.openstreetmap.org/api/0.6/relation/2919627 http://api.openstreetmap.org/api/0.6/relation/2919628 | grep node | awk '{print $3}' | uniq ...
14
votes
2answers
643 views

What did `uniq -t` do?

I have some old code from 2003 which uses -t option for uniq command. It throws an error since that option is probably not supported anymore. Here's the piece which uses the command: egrep -n "{ ...
2
votes
4answers
391 views

print parent directories full path of find output

what I want to do is to find all files based on some search query and get parents parent directory (../..) full path. For example find . -name "__openerp__.py" and then for each file execute ...
2
votes
3answers
293 views

I need data from a csv file which is not present in the second csv file

For example: I have a CSV file (first file ) with below data (vertically): 1 2 3 4 5 6 Second CSV with below data (vertically): 5 6 7 8 I want my resultant file (action needs to be performed ...
2
votes
3answers
106 views

get list of months using dseq from dateutils

how'd I get a listing of the months in the year (abbreviated) using dseq from dateutils? NOTE simply running: dseq 'jan' 'dec' -i '%b' -f '%b' returns a humongous list of months: piping the ...
2
votes
1answer
97 views

Sort doesn't use natural order

I have used uniq -c and sort extensively on other machines, and it always worked very well. Now, on my new Ubuntu (13.04) machine, I end up with a weird ordering: % cat queries.log | grep INSERT | ...
23
votes
4answers
11k views

What is the difference between “sort -u” and “sort | uniq”?

Everywhere I see someone needing to get a sorted, unique list, they always pipe to sort | uniq. I've never seen any examples where someone uses sort -u instead. Why not? What's the difference, and why ...
0
votes
1answer
442 views

How to use uniq Unix command and specify input and output file?

I'm confusing use of uniq command, can anyone explain this command and how to use it? How to use uniq without specifying any option? Any example would be helped and appreciated
1
vote
1answer
1k views

How to match 2 large file and print the difference in shell script

I have 2 files. File1 has 400k numerical records. e.g: 1 2 3 4 5 6 .. and so on File 2 also has 420k numerical records. e.g: 1 2 3 4 6 .. and so on Both these file are in unsorted manner. I want ...
4
votes
1answer
292 views

Why is uniq ignoring Unicode and lines with a single letter?

I'm trying to combine both the American and British dictionaries into one large dictionaries, and I'm trying to remove all the duplicates from the superset, but it seems that uniq is not outputting ...
3
votes
1answer
4k views

The simplest method to count lines matching specific patterns, including '0' if line is not found?

I have very big logs (several gigabytes per day), that can (but do not need to) contain specific lines. I have to count the number of occurences of every one of these lines on a daily basis. I have a ...
0
votes
1answer
3k views

Unique lines based on the first field

How can you find unique lines or duplicate lines in a file based on the content of the first field, or the first n characters ? The uniq command on OS X lacks the -w switch that is available under ...
10
votes
3answers
3k views

How can I remove duplicates in my .bash_history, preserving order?

I really enjoying using control+r to recursively search my command history. I've found a few good options I like to use with it: # ignore duplicate commands, ignore commands starting with a space ...
4
votes
3answers
5k views

Get lines with maximum values in the column using awk, uniq and sort

I have a file with next format 2011-12-01 user1 access1 2011-12-01 user1 access2 2011-12-01 user2 access2 2011-12-01 user4 access2 2011-12-02 user1 access1 2012-01-01 user3 access1 2012-01-01 user4 ...
11
votes
5answers
4k views

case-insensitive search of duplicate file-names

I there a way to find all files in a directory with duplicate filenames, regardless of the casing (upper-case and/or lower-case)?
1
vote
3answers
1k views

How let `sort | uniq -c` separate the number of occurrences by a tab charactor?

uniq -c separate the occurrence number by spaces, which is hard for to cut or awk to separate it out later. 1000_A1\tB1\n ___1_A2\tB2\n I can solve this problem using sed -r 's/^ *([0-9]+)/\1\t/' ...
7
votes
1answer
3k views

How to remove duplicate lines in a large multi-GB textfile?

My question is similar to this question but with a couple of different constraints: I have a large \n delimited wordlist -- one word per line. Size of files range from 2GB to as large as 10GB. I ...
4
votes
2answers
1k views

Where has my `uniq` or `sort -u` line gone, with some unicode characters

What is going on in the following code snippet? I'm not getting my expected output. I'd think it was a bug, but it happens for 2 different programs (uniq and sort), so I suspect it is something ...
18
votes
4answers
5k views

How to get only the unique results without having to sort data?

$ cat data.txt aaaaaa aaaaaa cccccc aaaaaa aaaaaa bbbbbb $ cat data.txt | uniq aaaaaa cccccc aaaaaa bbbbbb $ cat data.txt | sort | uniq aaaaaa bbbbbb cccccc $ The result that I need is to display ...