106
votes
How can I remove duplicates in my .bash_history, preserving order?
So I was looking for the same exact thing after being annoyed by duplicates, and found that if I edit my ~/.bash_profile or my ~/.bashrc with:
export HISTCONTROL=ignoreboth:erasedups
It does almost ...
- 1,173
100
votes
Accepted
Common lines between two files
Use comm -12 file1 file2 to get common lines in both files.
You may also needs your file to be sorted to comm to work as expected.
comm -12 <(sort file1) <(sort file2)
From man comm:
-1 ...
- 41.1k
37
votes
Accepted
What is the point of uniq -u and what does it do?
This ought to be easy to test:
$ cat file
1
2
3
3
4
4
$ uniq file
1
2
3
4
$ uniq -u file
1
2
In short, uniq with no options removes all but one instance of consecutively duplicated lines. The GNU ...
- 327k
30
votes
Accepted
Who killed my sort? or How to efficient count distinct values from a csv column
TL;DR: out-of-memory-killer or running out of disk space for temporary files kills sort. Recommendation: Use a different tool.
I've had a glance over GNU coreutils' sort.c right now¹. Your -S 1G just ...
- 25k
27
votes
Accepted
Use uniq to filter adjacent lines in pipeline
It's the usual behavior of many tools, inherited from default Standard C library behavior with functions manipulating I/O streams (fopen(3), fwrite(3) ...). This is documented in setvbuf(3) (or a ...
- 33.4k
26
votes
Accepted
Remove adjacent duplicate lines while keeping the order
uniq will do this for you:
$ uniq inputfile
Golgb1
Akna
Spata20
Golgb1
Akna
- 74.7k
20
votes
Accepted
Keeping unique rows based on information from 2 of three columns
There is a "famous" awk idiom for exactly this. You want to do:
awk '!seen[$1,$2]++' file
That creates an associative array "seen" with the 2 columns as the key. Use the post-increment operator so ...
- 85.1k
16
votes
How can I find the most frequent word in a .csv file, ignoring duplicates on each line?
With GNU grep or compatible:
$ grep -nEo '\w+' file.csv|sort -u|cut -d: -f2-|sort|uniq -c|sort -k1rn|head
2 blue
2 green
2 red
1 brown
1 yellow
- 533k
15
votes
Output lines with the same md5 sum
If your task is to find duplicate files, you could also use fdupes:
Searches the given path for duplicate files. Such files are found by comparing file sizes and MD5 signatures, followed by a ...
- 22.4k
14
votes
Accepted
Why doesn't "uniq --unique" remove all duplicate lines?
uniq requires the input to be sorted (from man uniq) if you want it to remove all duplicate lines:
DESCRIPTION
Filter adjacent matching lines from INPUT (or standard input), writing
to OUTPUT (or ...
- 238k
13
votes
Accepted
Difference between using `sort -u` and `sort | uniq -u`
sort -u and sort | uniq do produce the same output*: all of the lines in the input, exactly once each, in ascending order. That is the default behaviour of uniq.
uniq -u, on the other hand, asks to:
...
- 75.6k
12
votes
How can I remove duplicates in my .bash_history, preserving order?
This is an old post, but a perpetual issue for users who want to have multiple terminals open, and have the history synched between windows, but not duplicated.
My solution in .bashrc:
shopt -s ...
- 221
12
votes
What is the point of uniq -u and what does it do?
From uniq(1):
NAME
uniq - report or omit repeated lines
DESCRIPTION
...
With no options, matching lines are merged to the first occurrence.
...
-u, --unique
only print unique lines
...
- 13.1k
12
votes
Accepted
Output lines with the same md5 sum
If you’ve got GNU uniq, you can ask it to show all lines duplicating the first 32 characters¹:
find path -type f -exec md5sum {} + | sort | uniq -D -w32
The list needs to be sorted since uniq only ...
- 423k
11
votes
Efficiently merge / sort / unique large number of text files
A simple fix, works at least in Bash, since printf is builtin, and the command line argument limits don't apply to it:
printf "%s\0" * | xargs -0 cat | sort -u > /tmp/bla.txt
(echo * | xargs ...
- 135k
11
votes
Accepted
How can I find the most frequent word in a .csv file, ignoring duplicates on each line?
I would probably reach for perl
Use uniq from the List::Util module to de-duplicate each row.
Use a hash to count the resulting occurrences.
For example
perl -MList::Util=uniq -F, -lnE '
map { $h{...
- 79.6k
10
votes
Accepted
Unix - count unique IP addresses, sort them by most frequent and also sort them by IP when number of repetitions is same
If your sort can do a stable sort, e.g. GNU sort with the -s or --stable option, lines with fields unrelated to the sort keys will not be sorted by those fields when there are ties, but will stay in ...
- 25.1k
10
votes
Remove adjacent duplicate lines while keeping the order
Awk solution:
awk '$1 != name{ print }{ name = $1 }' file.txt
The output:
Golgb1
Akna
Spata20
Golgb1
Akna
- 29.9k
10
votes
Accepted
Using uniq on unicode text
The GNU implementation of uniq as found on Ubuntu, with -c, doesn't report counts of contiguous identical lines but counts of contiguous lines that sort the same¹.
Most international locales on GNU ...
- 533k
9
votes
Efficiently merge / sort / unique large number of text files
find . -maxdepth 1 -type f ! -name ".*" -exec cat {} + | sort -u -o /path/to/sorted.txt
This will concatenate all non-hidden regular files in the current directory and sort their combined contents (...
- 327k
9
votes
Accepted
Efficiently merge / sort / unique large number of text files
With GNU sort, and a shell where printf is built-in (all POSIX-like ones nowadays except some variants of pdksh):
printf '%s\0' * | sort -u --files0-from=- > output
Now, a problem with that is ...
- 533k
9
votes
Accepted
Identify unique records on CSV based on specific columns
The idiomatic awk answer is awk -F, '!seen[$3]++' file
That will print a line the first time a value is seen in the 3rd column.
- 85.1k
9
votes
cut command fields
It is because the unquoted `` backquote command substitution has removed an extra space between the model name and the : characters. Refer to the outputs without the grep to see the difference for ...
- 12.6k
9
votes
How can I find the most frequent word in a .csv file, ignoring duplicates on each line?
You can use awk with an associative array and a simple logic check.
awk -F, '
{split("", c); for (i=1; i<=NF; i++)
if (!c[$i]){c[$i]++;wds[$i]++}}
END{for (wd in wds) print wds[...
- 4,718
8
votes
How to find the number of unordered pairs from a list
It sounds like a good job for perl:
perl -F -lane '$count{join "", sort @F}++;
END{print "$count{$_} $_" for sort keys %count}' < your-file
- 533k
8
votes
Accepted
Less expensive version of `sort -n | uniq -c | sort -n`
With single awk process:
awk '{ a[$1]++ }END{ for(i in a) print a[i],i }' file
The output:
3 1.1.1.1
2 2.2.2.2
1 3.3.3.3
To output records sorted by number of occurrences in descending order use ...
- 29.9k
8
votes
Accepted
What does `-dD` mean to `uniq`?
TLDR Bottom line, they do nothing when used together; -dD is identical to -D.
Research
If you look at the case/switch logic of the uniq.c command you can see this first hand:
case 'd':
...
slm♦
- 366k
8
votes
How can I find the most frequent word in a .csv file, ignoring duplicates on each line?
Using awk:
awk -F , '
{
delete seen
for (i = 1; i <= NF; ++i) seen[$i]++ || ++count[$i]
}
END {
for (word in count) printf "%...
- 327k
8
votes
Print count number of unique pattern occurences after each occurence
You could use a simple awk command for the whole task:
awk '/Exception/{a[$0]++} END {for (x in a) print x,a[x]}' file | sort -nk2
Output
ExceptionD 1
ExceptionA 2
ExceptionB 3
The order for an awk ...
- 7,962
8
votes
Accepted
Why can't I get a unique list of shells?
It's because how uniq works, from the man:
Note: 'uniq' does not detect repeated lines unless they are
adjacent. You may want to sort the input first, or use 'sort -u'
without 'uniq'.
So, ...
- 12.1k
Only top scored, non community-wiki answers of a minimum length are eligible