Questions tagged [uniq]
The uniq tag has no usage guidance.
173
questions
169
votes
5
answers
260k
views
What is the difference between "sort -u" and "sort | uniq"?
Everywhere I see someone needing to get a sorted, unique list, they always pipe to sort | uniq. I've never seen any examples where someone uses sort -u instead. Why not? What's the difference, and why ...
- 5,772
104
votes
13
answers
49k
views
How can I remove duplicates in my .bash_history, preserving order?
I really enjoying using control+r to recursively search my command history. I've found a few good options I like to use with it:
# ignore duplicate commands, ignore commands starting with a space
...
- 44.6k
60
votes
4
answers
76k
views
How to get only the unique results without having to sort data?
$ cat data.txt
aaaaaa
aaaaaa
cccccc
aaaaaa
aaaaaa
bbbbbb
$ cat data.txt | uniq
aaaaaa
cccccc
aaaaaa
bbbbbb
$ cat data.txt | sort | uniq
aaaaaa
bbbbbb
cccccc
$
The result that I need is to display ...
- 34.6k
46
votes
2
answers
106k
views
Common lines between two files [duplicate]
I have the following code that I run on my Terminal.
LC_ALL=C && grep -F -f genename2.txt hg38.hgnc.bed > hg38.hgnc.goi.bed
This doesn't give me the common lines between the two files. ...
- 653
44
votes
4
answers
10k
views
How is uniq not unique enough that there is also uniq --unique?
Here are commands on a random file from pastebin:
wget -qO - http://pastebin.com/0cSPs9LR | wc -l
350
wget -qO - http://pastebin.com/0cSPs9LR | sort -u | wc -l
287
wget -qO - http://pastebin.com/...
- 561
20
votes
8
answers
15k
views
case-insensitive search of duplicate file-names
I there a way to find all files in a directory with duplicate filenames, regardless of the casing (upper-case and/or lower-case)?
- 893
19
votes
3
answers
23k
views
Uniq won't remove duplicate
I was using the following command
curl -silent http://api.openstreetmap.org/api/0.6/relation/2919627 http://api.openstreetmap.org/api/0.6/relation/2919628 | grep node | awk '{print $3}' | uniq
when ...
- 507
18
votes
1
answer
13k
views
How to remove duplicate lines in a large multi-GB textfile?
My question is similar to this question but with a couple of different constraints:
I have a large \n delimited wordlist -- one word per line. Size of
files range from 2GB to as large as 10GB.
I ...
- 283
17
votes
12
answers
7k
views
Delete duplicate lines pairwise?
I encountered this use case today. It seems simple at first glance, but fiddling around with sort, uniq, sed and awk revealed that it's nontrivial.
How can I delete all pairs of duplicate lines? In ...
- 35.4k
16
votes
5
answers
4k
views
Remove adjacent duplicate lines while keeping the order
I have a file with one column with names that repeat a number of times each. I want to condense each repeat into one, while keeping any other repeats of the same name that are not adjacent to other ...
- 559
15
votes
8
answers
4k
views
How can I find the most frequent word in a .csv file, ignoring duplicates on each line?
I need to find the 10 most frequent words in a .csv file.
The file is structured so that each line contains comma-separated words. If the same word is repeated more than once in the same line, it ...
- 370
15
votes
2
answers
4k
views
What did `uniq -t` do?
I have some old code from 2003 which uses -t option for uniq command. It throws an error since that option is probably not supported anymore.
Here's the piece which uses the command:
egrep -n "{ ...
- 875
14
votes
3
answers
2k
views
What is the point of uniq -u and what does it do? [duplicate]
uniq seems to do something different than uniq -u, even though the description for both is "only unique lines".
What's the difference here, what do they do?
- 159
14
votes
2
answers
4k
views
Who killed my sort? or How to efficient count distinct values from a csv column
I'm doing some processing trying to get how many different lines in a file containing 160,353,104 lines. Here is my pipeline and stderr output.
$ tail -n+2 2022_place_canvas_history.csv | cut -d, -f2 |...
- 213
12
votes
5
answers
16k
views
How to create an array of unique elements from a string/array in bash?
If I have a string "1 2 3 2 1" - or an array [1,2,3,2,1] - how can I select the unique values, i.e.
"1 2 3 2 1" produces "1 2 3"
or
[1,2,3,2,1] produces [1,2,3]
Similar to uniq but uniq seems to ...
- 41.3k
11
votes
1
answer
933
views
Use uniq to filter adjacent lines in pipeline
I'm trying to monitor theme changes using this command:
dbus-monitor --session "interface='org.freedesktop.portal.Settings', member=SettingChanged" | grep -o "uint32 ."
Output ...
- 235
11
votes
4
answers
13k
views
sum pair of columns based on matching fields
I have a large file in the following format:
2 1019 0 12
2 1019 3 0
2 1021 0 2
2 1021 2 0
2 1022 4 5
2 1030 0 1
2 1030 5 0
2 1031 4 4
If the values in column 2 match, I want to sum the values ...
- 113
10
votes
2
answers
3k
views
Where has my `uniq` or `sort -u` line gone, with some unicode characters
What is going on in the following code snippet? I'm not getting my expected output.
I'd think it was a bug, but it happens for 2 different programs (uniq and sort), so I suspect it is something ...
- 32.5k
10
votes
2
answers
38k
views
sort and uniq in awk
I know there are "sort" and "uniq" out there, however, today's question is about how to utilise AWK to do that kind of a job. Say if I have a list of anything really (ips, names, or numbers) and I ...
- 199
9
votes
2
answers
8k
views
Keeping unique rows based on information from 2 of three columns
Suppose you have a file like this:
NW_006521251.1 428 84134
NW_006521251.1 511 84135
NW_006521038.1 202 84155
NW_006521038.1 1743 84153
NW_006521038.1 1743 84154
NW_006520495.1 198 84159
...
- 559
8
votes
5
answers
5k
views
Efficiently merge / sort / unique large number of text files
I am trying a naive:
$ cat * | sort -u > /tmp/bla.txt
which fails with:
-bash: /bin/cat: Argument list too long
So in order to avoid a silly solution like (creates an enormous temporary file):
...
- 2,718
8
votes
3
answers
7k
views
Fastest `uniq` tool in linux
I have large text file (1.5 G),
I want to know what the fastest and more reliable tool in Linux is.
I usualy use:
awk '!x[$0]++' file.txt
But when I use htop command I see my memory usage is ...
- 972
7
votes
4
answers
568
views
Groupwise uniq command?
I am searching for a command to get from a file in this format:
hello 32
hello 67
hi 2
ho 1212
ho 1390
ho 3000
To this format (deduplicate by taking the last row of a "group"):
hello 67
...
- 673
7
votes
1
answer
1k
views
Why can uniq ignore leading characters but not ending characters? [closed]
The POSIX-specified command line tool uniq is designed to "report or filter out repeated lines in a file." Aside from use cases involving the full lines for comparison, there are two options for ...
- 35.4k
6
votes
3
answers
2k
views
Using uniq on unicode text
I want to remove duplicate lines from a file with words of Syriac script. The source file has 3 lines, 1st and 3rd are identical.
$ cat file.txt
ܐܒܘܢ
ܢܗܘܐ
ܐܒܘܢ
When I use sort and uniq, the result ...
- 163
6
votes
4
answers
542
views
How to find the number of unordered pairs from a list
I want to count the number of unordered pairs from a list.
In other words, I have a list:
ab
ba
ac
bc
and I want to display:
2 ab
1 ac
1 bc
- 77
6
votes
1
answer
18k
views
Difference between using `sort -u` and `sort | uniq -u`
I came into a level while exploring the bash that consisted of finding the line of text that occurs only once in a certain file.
Why is the output of the sort -u file command different from the ...
- 175
6
votes
2
answers
10k
views
Count & sum no. of occurrences of each unique line in a file [duplicate]
I think the answer to this problem might be some sort of variant of the uniq function that allows you to count the number of times each line appears in a file:
sort file.txt | uniq -c
The problem ...
- 141
6
votes
3
answers
9k
views
Unix - count unique IP addresses, sort them by most frequent and also sort them by IP when number of repetitions is same
I have a list of IP addresses in file:
72.204.55.250
72.204.55.250
72.204.55.250
72.204.55.250
72.204.55.250
96.41.51.202
208.115.113.91
178.137.94.166
178.137.94.166
208.115.113....
- 163
6
votes
1
answer
2k
views
`uniq` is not realtime when piped
When I run
for i in {1..2000}; do sleep 0.1 && echo $(netstat -s | grep -i syns); done | uniq
i can see in realtime when expression $(netstat -s | grep -i syns) changes.
But when I want to ...
- 533
6
votes
1
answer
2k
views
Why is uniq ignoring Unicode and lines with a single letter?
I'm trying to combine both the American and British dictionaries into one large dictionaries, and I'm trying to remove all the duplicates from the superset, but it seems that uniq is not outputting ...
- 864
5
votes
2
answers
4k
views
Why doesn't "uniq --unique" remove all duplicate lines?
Running
printf "lol\nlol\nfoo\n\n\n\n\nbar\nlol\nlol\nfoo\nlol\nfoo" | uniq --unique
prints
foo
bar
foo
lol
foo
Why is foo printed three times? Shouldn't uniq --unique remove them?
Also, ...
- 585
5
votes
4
answers
6k
views
How to find duplicate lines in a text file, while some may be commented out or have different tokens at the beginning?
I have a text file with lines that are a mixture like this:
### Comments
# Comments
86.242.200.81 banana.domain.net # comment
86.242.200.3 orange.domain.net
31.28.225.81 monkey....
- 1,171
5
votes
3
answers
12k
views
Get lines with maximum values in the column using awk, uniq and sort
I have a file with next format
2011-12-01 user1 access1
2011-12-01 user1 access2
2011-12-01 user2 access2
2011-12-01 user4 access2
2011-12-02 user1 access1
2012-01-01 user3 access1
2012-01-01 user4 ...
- 1,095
5
votes
2
answers
1k
views
Print count number of unique pattern occurences after each occurence
Supposing I have a log file containing thrown Exceptions:
ExceptionA
loggedFunctionCall
ExceptionB
ExceptionA
loggedFunctionCall
ExceptionD
ExceptionB
loggedFunctionCall
ExceptionB
I want to count ...
- 159
5
votes
1
answer
353
views
cut command fields
I noticed these two different fields behaviors using cut command:
bash:~$ var=`cat /proc/cpuinfo | grep 'model name' | uniq | cut -d ' ' -f 3,4,5,6,7,8 `
echo $var
outputs
Intel(R) Core(TM) i7-...
- 419
5
votes
1
answer
5k
views
Sort doesn't use natural order
I have used uniq -c and sort extensively on other machines, and it always worked very well.
Now, on my new Ubuntu (13.04) machine, I end up with a weird ordering:
% cat queries.log | grep INSERT | ...
- 1,162
5
votes
1
answer
427
views
Why does uniq think あい and いあ are the same?
I'm trying to find duplicates in files, by using:
sort myfile | uniq -d
I noticed that uniq seems to dislike Japanese characters for some reason. For example, if I have a file:
あい
いあ
Then
sort ...
- 178
5
votes
1
answer
206
views
Why does uniq drop korean string?
I have a CSV file containing 220k names that I want to batch import into a graph database. The batch importer imposes the restriction that each node must occur only once so I want to remove duplicates ...
- 151
4
votes
3
answers
4k
views
Remove nearly duplicate lines
I've got a knotty problem that I can't figure out how to solve.
I have a text file containing a few million lines of text. Basically I want to run uniq, but with a twist: If two lines are identical ...
- 5,704
4
votes
4
answers
4k
views
Find duplicates by column in a file
I have an input file containing file paths and their md5sum, separated by semicolons and sorted by md5 hashes :
/media/LaCie/Images/recup2/recup_dir.1/f1295328.jpg;0080ececd3da8533f5d11e449cf73287
/...
- 141
4
votes
3
answers
113
views
Script optimisation to find duplicates filename in huge CSV
I have several CSV file from 1MB to 6GB generated by inotify script with a list of events formatted as is:
timestamp;fullpath;event;size.
Those files are formatted like this :
timestamp;fullpath;...
- 9,445
4
votes
1
answer
8k
views
The simplest method to count lines matching specific patterns, including '0' if line is not found?
I have very big logs (several gigabytes per day), that can (but do not need to) contain specific lines. I have to count the number of occurences of every one of these lines on a daily basis.
I have a ...
- 1,646
4
votes
2
answers
1k
views
Sort and remove duplicates in place and recursively
I would like to sort all files in all subdirectories in place. I have 65536 files in 256 directories, every file containing one word a line, and each file contains duplicates.
What I would like is to ...
- 71
4
votes
2
answers
967
views
Remove duplicate lines with a twist [duplicate]
Okay so I want remove duplicate lines but it's a bit more complicated than that..
I have a file named users.txt, example of file is:
users:[email protected]
users1:[email protected]
Now due to a bug ...
- 43
4
votes
3
answers
3k
views
uniq showing duplicate lines
$ grep home american.txt | sort | uniq
A home and a country should leave us no more! ...
- 241
4
votes
1
answer
150
views
Generalization of uniq
The uniq(1) command can be viewed as a tiny map/reduce-style program:
map consecutive, identical input lines into groups;
reduce the list of lines (which are all identical) to a single line.
Are ...
- 673
4
votes
5
answers
5k
views
How to let `sort | uniq -c` separate the number of occurrences by a tab character?
uniq -c separates the occurrence number by spaces, which is hard for cut or awk to separate it out later.
1000_A1\tB1\n
___1_A2\tB2\n
I can solve this problem using sed -r 's/^ *([0-9]+)/\1\t/' to ...
- 6,551
4
votes
2
answers
2k
views
uniq and bash for loop not writing to stdout before stdin closing (for one-line website visitor notification system)
I'm trying to trigger a beep on the PC speaker for every unique visitor of a website.
After some brainstorming, it seemed to be possible with one line:
for e in `ssh me@mymachine "tail -n 1 -f /var/...
- 141
3
votes
2
answers
799
views
Output lines with the same md5 sum
I have a script like this
find path -type f -exec md5sum {} +'
It has this conclusion
/tmp
❯ find $pwd -type f -exec md5sum {} +
\a7c8252355166214d1f6cd47db917226 ./guess.bash
...
- 51