Skip to main content
103 votes
Accepted

Common lines between two files

Use comm -12 file1 file2 to get common lines in both files. You may also needs your file to be sorted to comm to work as expected. comm -12 <(sort file1) <(sort file2) From man comm: -1 ...
αғsнιη's user avatar
  • 41.5k
37 votes
Accepted

What is the point of uniq -u and what does it do?

This ought to be easy to test: $ cat file 1 2 3 3 4 4 $ uniq file 1 2 3 4 $ uniq -u file 1 2 In short, uniq with no options removes all but one instance of consecutively duplicated lines. The GNU ...
Kusalananda's user avatar
  • 339k
30 votes
Accepted

Who killed my sort? or How to efficient count distinct values from a csv column

TL;DR: out-of-memory-killer or running out of disk space for temporary files kills sort. Recommendation: Use a different tool. I've had a glance over GNU coreutils' sort.c right now¹. Your -S 1G just ...
Marcus Müller's user avatar
27 votes
Accepted

Use uniq to filter adjacent lines in pipeline

It's the usual behavior of many tools, inherited from default Standard C library behavior with functions manipulating I/O streams (fopen(3), fwrite(3) ...). This is documented in setvbuf(3) (or a ...
A.B's user avatar
  • 37.2k
26 votes
Accepted

Remove adjacent duplicate lines while keeping the order

uniq will do this for you: $ uniq inputfile Golgb1 Akna Spata20 Golgb1 Akna
DopeGhoti's user avatar
  • 76.9k
20 votes
Accepted

Keeping unique rows based on information from 2 of three columns

There is a "famous" awk idiom for exactly this. You want to do: awk '!seen[$1,$2]++' file That creates an associative array "seen" with the 2 columns as the key. Use the post-increment operator so ...
glenn jackman's user avatar
16 votes

How can I find the most frequent word in a .csv file, ignoring duplicates on each line?

With GNU grep or compatible: $ grep -nEo '\w+' file.csv|sort -u|cut -d: -f2-|sort|uniq -c|sort -k1rn|head 2 blue 2 green 2 red 1 brown 1 yellow
Stéphane Chazelas's user avatar
15 votes
Accepted

Why doesn't "uniq --unique" remove all duplicate lines?

uniq requires the input to be sorted (from man uniq) if you want it to remove all duplicate lines: DESCRIPTION Filter adjacent matching lines from INPUT (or standard input), writing to OUTPUT (or ...
terdon's user avatar
  • 244k
15 votes

Output lines with the same md5 sum

If your task is to find duplicate files, you could also use fdupes: Searches the given path for duplicate files. Such files are found by comparing file sizes and MD5 signatures, followed by a ...
pLumo's user avatar
  • 22.7k
13 votes
Accepted

Output lines with the same md5 sum

If you’ve got GNU uniq, you can ask it to show all lines duplicating the first 32 characters¹: find path -type f -exec md5sum {} + | sort | uniq -D -w32 The list needs to be sorted since uniq only ...
Stephen Kitt's user avatar
12 votes

How can I remove duplicates in my .bash_history, preserving order?

This is an old post, but a perpetual issue for users who want to have multiple terminals open, and have the history synched between windows, but not duplicated. My solution in .bashrc: shopt -s ...
smilingfrog's user avatar
12 votes
Accepted

Difference between using `sort -u` and `sort | uniq -u`

sort -u and sort | uniq do produce the same output*: all of the lines in the input, exactly once each, in ascending order. That is the default behaviour of uniq. uniq -u, on the other hand, asks to: ...
Michael Homer's user avatar
12 votes

What is the point of uniq -u and what does it do?

From uniq(1): NAME uniq - report or omit repeated lines DESCRIPTION ... With no options, matching lines are merged to the first occurrence. ... -u, --unique only print unique lines ...
Stewart's user avatar
  • 14.2k
11 votes

Efficiently merge / sort / unique large number of text files

A simple fix, works at least in Bash, since printf is builtin, and the command line argument limits don't apply to it: printf "%s\0" * | xargs -0 cat | sort -u > /tmp/bla.txt (echo * | xargs ...
ilkkachu's user avatar
  • 141k
11 votes
Accepted

How can I find the most frequent word in a .csv file, ignoring duplicates on each line?

I would probably reach for perl Use uniq from the List::Util module to de-duplicate each row. Use a hash to count the resulting occurrences. For example perl -MList::Util=uniq -F, -lnE ' map { $h{...
steeldriver's user avatar
  • 81.8k
10 votes

Remove adjacent duplicate lines while keeping the order

Awk solution: awk '$1 != name{ print }{ name = $1 }' file.txt The output: Golgb1 Akna Spata20 Golgb1 Akna
RomanPerekhrest's user avatar
10 votes
Accepted

Using uniq on unicode text

The GNU implementation of uniq as found on Ubuntu, with -c, doesn't report counts of contiguous identical lines but counts of contiguous lines that sort the same¹. Most international locales on GNU ...
Stéphane Chazelas's user avatar
9 votes

Efficiently merge / sort / unique large number of text files

find . -maxdepth 1 -type f ! -name ".*" -exec cat {} + | sort -u -o /path/to/sorted.txt This will concatenate all non-hidden regular files in the current directory and sort their combined contents (...
Kusalananda's user avatar
  • 339k
9 votes
Accepted

Efficiently merge / sort / unique large number of text files

With GNU sort, and a shell where printf is built-in (all POSIX-like ones nowadays except some variants of pdksh): printf '%s\0' * | sort -u --files0-from=- > output Now, a problem with that is ...
Stéphane Chazelas's user avatar
9 votes
Accepted

Identify unique records on CSV based on specific columns

The idiomatic awk answer is awk -F, '!seen[$3]++' file That will print a line the first time a value is seen in the 3rd column.
glenn jackman's user avatar
9 votes

cut command fields

It is because the unquoted `` backquote command substitution has removed an extra space between the model name and the : characters. Refer to the outputs without the grep to see the difference for ...
Inian's user avatar
  • 12.9k
9 votes

How can I find the most frequent word in a .csv file, ignoring duplicates on each line?

You can use awk with an associative array and a simple logic check. awk -F, ' {split("", c); for (i=1; i<=NF; i++) if (!c[$i]){c[$i]++;wds[$i]++}} END{for (wd in wds) print wds[...
bu5hman's user avatar
  • 4,771
9 votes

Print count number of unique pattern occurences after each occurence

You could use a simple awk command for the whole task: awk '/Exception/{a[$0]++} END {for (x in a) print x,a[x]}' file | sort -nk2 Output ExceptionD 1 ExceptionA 2 ExceptionB 3 The order for an awk ...
thanasisp's user avatar
  • 8,252
8 votes

How to find the number of unordered pairs from a list

It sounds like a good job for perl: perl -F -lane '$count{join "", sort @F}++; END{print "$count{$_} $_" for sort keys %count}' < your-file
Stéphane Chazelas's user avatar
8 votes
Accepted

Less expensive version of `sort -n | uniq -c | sort -n`

With single awk process: awk '{ a[$1]++ }END{ for(i in a) print a[i],i }' file The output: 3 1.1.1.1 2 2.2.2.2 1 3.3.3.3 To output records sorted by number of occurrences in descending order use ...
RomanPerekhrest's user avatar
8 votes
Accepted

What does `-dD` mean to `uniq`?

TLDR Bottom line, they do nothing when used together; -dD is identical to -D. Research If you look at the case/switch logic of the uniq.c command you can see this first hand: case 'd': ...
slm's user avatar
  • 372k
8 votes

How can I find the most frequent word in a .csv file, ignoring duplicates on each line?

Using awk: awk -F , ' { delete seen for (i = 1; i <= NF; ++i) seen[$i]++ || ++count[$i] } END { for (word in count) printf "%...
Kusalananda's user avatar
  • 339k
8 votes
Accepted

Why can't I get a unique list of shells?

It's because how uniq works, from the man: Note: 'uniq' does not detect repeated lines unless they are adjacent. You may want to sort the input first, or use 'sort -u' without 'uniq'. So, ...
schrodingerscatcuriosity's user avatar
7 votes
Accepted

`uniq` is not realtime when piped

Perhaps you only have to change the buffering: # buffering totally off stdbuf -i0 -o0 -e0 command # buffering per line only stdbuf -oL -eL command It works for me here... your results may vary. If ...
Peter's user avatar
  • 1,247
7 votes

How to find the number of unordered pairs from a list

With gawk: gawk -F '' '{ print ($1 > $2) ? $2$1 : $1$2 }' | sort | uniq -c Each field matches one character (-F ''). We just invert the characters if the first is greater than the second ...
xhienne's user avatar
  • 17.9k

Only top scored, non community-wiki answers of a minimum length are eligible