Questions tagged [uniq]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
3 answers
54 views

de-duplicate list but group parts of it

I am compiling some access rules from failed logins and after some piping I arrived at this: cat <<INPUT | sort -k 3,3 --unique Deny from 13.42.98.142 # demo Deny from 13.42.98.142 # test Deny ...
user avatar
14 votes
2 answers
4k views

Who killed my sort? or How to efficient count distinct values from a csv column

I'm doing some processing trying to get how many different lines in a file containing 160,353,104 lines. Here is my pipeline and stderr output. $ tail -n+2 2022_place_canvas_history.csv | cut -d, -f2 |...
user avatar
  • 193
3 votes
5 answers
319 views

find duplicate 1st field and concat its values in single line

I have a file that has entries in key: value format like the below: cat data.txt name: 'tom' tom_age: '31' status_tom_mar: 'yes' school: 'anne' fd_year_anne: '1987' name: 'hmz' hmz_age: '21' ...
user avatar
  • 297
-1 votes
1 answer
31 views

Skip line from console if equal than line before, adding count (in realtime)

Using uniq it is possible to filter out sequential duplicate lines. while (true) do echo 1; echo 2; echo 2; echo 1; sleep 1; done | uniq becomes: 1 2 1 Is there a way to have duplicated sequential ...
user avatar
3 votes
2 answers
691 views

Output lines with the same md5 sum

I have a script like this find path -type f -exec md5sum {} +' It has this conclusion /tmp ❯ find $pwd -type f -exec md5sum {} + \a7c8252355166214d1f6cd47db917226 ./guess.bash ...
user avatar
1 vote
4 answers
127 views

uniq - how to ignore IP address?

I have the following: cat /var/log/example.log | cut -d ' ' -f 3,4,5,6 | sort -u | uniq The result has several thousand lines but I would like to cut this down to show the actual activity and to do ...
user avatar
4 votes
2 answers
398 views

Sort and remove duplicates in place and recursively

I would like to sort all files in all subdirectories in place. I have 65536 files in 256 directories, every file containing one word a line, and each file contains duplicates. What I would like is to ...
user avatar
  • 71
4 votes
2 answers
2k views

Why doesn't "uniq --unique" remove all duplicate lines?

Running printf "lol\nlol\nfoo\n\n\n\n\nbar\nlol\nlol\nfoo\nlol\nfoo" | uniq --unique prints foo bar foo lol foo Why is foo printed three times? Shouldn't uniq --unique remove them? Also, ...
user avatar
1 vote
3 answers
467 views

How to sort uniq values from a text file? [duplicate]

I have a text file as shown below: A f1 B f2 A f3 B f4 B f5 I want to sort it based on first column values and keep it in separate files. Desired output: A.txt: A f1 A f3 B.txt: B f2 B f4 B f5 ...
user avatar
2 votes
4 answers
701 views

How can I remove duplicated tags based on specific attribute name in an XML file?

How can I remove duplicate line based on "groupName" and keep the line with directoryId="1"? <Group id="123" groupName="ABC" lowerGroupName="abc" ...
user avatar
1 vote
4 answers
217 views

How do I clean up a log file to show consecutive repeated lines only once, with a number indicating how many times that line repeated?

Current logs: 18:56:54 Info: Starting 18:56:55 Error: timed out 18:56:56 Error: timed out 18:56:57 Error: timed out 18:56:58 Info: reconnected 18:56:59 Error: timed out Desired output: 18:56:54 Info: ...
user avatar
3 votes
4 answers
650 views

How to remove leading spaces in multiple lines based on indent in first line?

Processing a text line-by-line removing leading spaces on each line is easy: $ LC_ALL=C git ls-files | sed -nE 's:^.*(\.[^./]+)$:\1:p' \ | sort | uniq -c | sort -snr > lines # create example &...
user avatar
  • 333
3 votes
2 answers
440 views

Why can't I get a unique list of shells? [duplicate]

Trying to get a unique list of shells on my system. When I run this command: cat /etc/passwd | cut -d ':' -f 7 | uniq I get: /bin/bash /bin/sync /sbin/shutdown /sbin/halt /bin/bash I can't ...
user avatar
1 vote
3 answers
186 views

Remove duplicated lines containing specific text

I have a big text file containing a million lines. I would like to find identical lines that match my specific text and leave the first occurrence intact. Any ideas? So the algorithm should roughly be ...
user avatar
  • 11
5 votes
2 answers
562 views

Print count number of unique pattern occurences after each occurence

Supposing I have a log file containing thrown Exceptions: ExceptionA loggedFunctionCall ExceptionB ExceptionA loggedFunctionCall ExceptionD ExceptionB loggedFunctionCall ExceptionB I want to count ...
user avatar
14 votes
3 answers
2k views

What is the point of uniq -u and what does it do? [duplicate]

uniq seems to do something different than uniq -u, even though the description for both is "only unique lines". What's the difference here, what do they do?
user avatar
0 votes
1 answer
65 views

Linux uniq: how to uniq the list ignore different remark

Original data (abc.csv): 8|AAAAA_001|0| 8|AAAAA_002|0| 8|AAAAA_003|0| 8|AAAAA_004|0| 8|AAAAA_005|0|AAAAA_005 8|AAAAA_006|0| 9|BBBBB_001|0| 9|BBBBB_002|0| 9|BBBBB_003|0|BBBBB_003 9|BBBBB_004|0| 9|...
user avatar
1 vote
2 answers
107 views

How to remove duplicate paragraphs

Given a file with paragraphs (group of lines separated by a blank line). It is guaranteed that the line breaks within equivalent paragraphs are conserved. Is there a way of using uniq to remove the ...
user avatar
0 votes
2 answers
95 views

How do I find the duplicate lines in a file, output them with the filename, and do this for an entire directory?

I have few files under directory and need to find duplicate lines in each file and display the file name to identify which file has duplicate. I tried: for i in *.*;do sort $i | uniq -d ; done It ...
user avatar
-1 votes
1 answer
38 views

print the count and the line after search based on pattern

I have lines like this in multiple log files FILE 1: Line 203:Caused by :arithmatic overflow exception Line 204:at com.abc.java FILE 2: Line 105:Caused by : ???String out of bound exception Line ...
user avatar
3 votes
3 answers
212 views

Sort -u (or sort | uniq) on a couple of columns but retain the info in a third column, and append it to the retained row?

I'm trying to take a table like this: a b hello a b goodbye g g test a c I say to collapse the table down to one row per unique entry on column 1 and 2, but without ...
user avatar
  • 33
-2 votes
1 answer
53 views

To count the number of occurrences of first field in text file [closed]

i require to count the number of occurrences of first field in txt file and to print output file as two column file having first field of input file & no. of occurrences of fist field in input ...
user avatar
0 votes
1 answer
50 views

How to use a series of commands on vim

Hi guys in the terminal I use the command cat file.txt | sort | uniq -c . First I would like to use that command on vim and save the output to another file without the output being print when I run ...
user avatar
  • 15
15 votes
8 answers
3k views

How can I find the most frequent word in a .csv file, ignoring duplicates on each line?

I need to find the 10 most frequent words in a .csv file. The file is structured so that each line contains comma-separated words. If the same word is repeated more than once in the same line, it ...
user avatar
  • 370
0 votes
2 answers
152 views

uniq -i is does not ignore case-sensitive in non-Ascii characters

I have a txt: a.txt A a B b Ş ş I try this command and I get false output: $ uniq -ic a.txt 2 A 2 B 1 Ş 1 ş How can I solve the non-ascii character problem ...
user avatar
  • 84
2 votes
2 answers
218 views

How can I remove blocks of duplicated lines with uniq?

Is there anyway to use uniq (or similar) to filter/remove sets of repeating lines from log type output? I am debugging an MPI code where multiple processors often print the same exact output. uniq ...
user avatar
  • 31
0 votes
4 answers
63 views

How do I print lines for the first appearance of a unique value in a 2-column file?

I have a small snippet of a file I'm working with: ENSDARG00000032737 ENSDARP00000120731 ENSDARG00000032737 ENSDARP00000049290 ENSDARG00000061051 ENSDARP00000081062 ENSDARG00000061051 ...
user avatar
0 votes
1 answer
66 views

Bash command to count lines with matching substrings in two different positions

I'm trying to get some trivial statistics on some debug output. Each debug line is of the form(class name)(delimiter 1)(object ID)(delimiter 2)(method name)(delimiter 3)(log message) I want to get a ...
user avatar
0 votes
2 answers
772 views

Remove all duplicate files in subdirectories in main directory

I am new to Linux and was wondering how I would go about deleting all duplicates with same file name but only keep one original? For example I have the following: dir_new_001 = rdt.txt dir_new_002 = ...
user avatar
2 votes
3 answers
2k views

finding all non-unique lines in a file

I'm trying to use uniq to find all non-unique lines in a file. By non-unique, I mean any line that I have already seen on the previous line. I thought that the "-D" option would do this: -D ...
user avatar
  • 474
0 votes
2 answers
101 views

Sort command changes string output

I stumbled on this anomaly regarding the sort command and I can't find an explamation to why it happens - running the following command: cat ~/.bash_history | uniq -c | sort -h -r | head Should ...
user avatar
  • 1
0 votes
3 answers
204 views

Delete line if other line starts with current line (column wise)

I've got a file input.txt in the following tab delimited format: aaaa bbbb aaaa bbbb c aaaa bbbb c dd aaaa bbbb cc aaaa bbbb x aaaa bbbb xx dddd eeee dddd ...
user avatar
  • 17
-2 votes
3 answers
382 views

How to count the number of occurrences of all strings in the file while getting the average of each string

I am trying to count the number of occurrences without specifying a specific string, just using the output of the cut command to print the number of each repeated string using grep. I then want to get ...
user avatar
  • 13
1 vote
2 answers
1k views

Bash - sort and uniq on array

This is a similar question to this one I want to do the word count but this time, using an array. For example, I have the following IPs inside a bash array called IPS. IPS=("1.1.1.1" "5.5.5.5" "3.3....
user avatar
  • 121
0 votes
3 answers
662 views

change format of uniq -c command

I want to get the unique words and its count from a file. When I run the bellow command, sort words.txt | uniq -c 2 america 4 and 1 england 1 file 1 for 1 place But I want the ...
user avatar
  • 143
1 vote
2 answers
2k views

Difference between cat file.txt | sort -u and cat file.txt | uniq

I want to know the difference between sort -u and uniq. I tried below command to get the count for the unique strings in a file , sort -u and uniq provides the same output from my knowledge. Then why ...
user avatar
2 votes
3 answers
208 views

Remove duplicate lines from file that are in a different order

My file is like this: alice, bob bob, cat cat, dennis cat, bob dennis, alice I want to remove lines where same words have been repeated in reverse order. In this example, bob, cat and cat, bob are ...
user avatar
3 votes
2 answers
118 views

Remove duplicate values that aren't on identical lines

So I have a set of text that contains both the file names and an associated number. Currently it looks like this: RR0.out -1752.142111 RR1.out -1752.141887 RR2.out -1752.142111 RR3.out -...
user avatar
0 votes
2 answers
3k views

How do I count the occurrences in a list, and then sort by descending occurrence?

I have a list with 1601 lines, where on each line, I am interested in counting the # of times different values appear for fields 5. Here's what I have so far: awk '{print$5}' FILE | sort | uniq -c ...
user avatar
  • 275
-1 votes
2 answers
136 views

How can i remove lines from a file when a string appears on multiple lines?

I have a file that has 2 columns like the following: apple pear banana pizza spoon fork pizza plate sausage egg If a word appears on multiple lines i want to delete all lines that the repeating word ...
user avatar
  • 73
5 votes
1 answer
160 views

Why does uniq drop korean string?

I have a CSV file containing 220k names that I want to batch import into a graph database. The batch importer imposes the restriction that each node must occur only once so I want to remove duplicates ...
user avatar
0 votes
2 answers
2k views

Show line numbers of a file after "cut -d ";" -f3 |uniq -d"

I am trying to make a simple command that can show me the duplicate data from one specific column and also give me the original line number. Example of file: JENNIE;30;DOCTOR;F SARA;26;POLICE;F ...
user avatar
2 votes
2 answers
9k views

Uniq Descending Sort in a column

In the given data is it possible to uniq sort and print only the top hits against each region? Given Data aza1 18 bcn1 16 sat2 12 lcy2 12 fra1 12 aza1 12 bcn1 10 sat2 8 lcy2 9 fra1 13 aza1 21 bcn1 2 ...
user avatar
2 votes
1 answer
130 views

uniq -c Equivalent for Groups of Lines of Arbitrary Count

I've got a file of ~1-2 million lines that I'm trying to reduce down by counting duplicate groups of lines, preserving order. uniq -c works okay : $ perl -E 'say for (("foo") x 4, ("bar") x 4, "baz"...
user avatar
  • 413
0 votes
1 answer
33 views

delete rows with duplications in first column in bash

I have a tab delimited .txt file and want to delete rows that have duplications in the first column. If this can be done without sort, that would be amazing, but I've kind of come to terms with that I ...
user avatar
0 votes
2 answers
82 views

How do I get a shell script to remove duplicates in a text file, based on the 11th 21st columns?

How do I get a shell script to remove duplicates in a text file, based on the 11th 21st columns? Sample file: Header:0000000000000001457854500000 XP 12345678912yeyeyeyeeye 0000003 XP ...
user avatar
  • 1
1 vote
4 answers
506 views

remove duplicate and replace it with empty space

I have a table as of below fruits shopname Apple x1 orange x1 banana x2 Apple x3 orange x2 banana x3 I want to group all rows based on column 1 and replace the duplicates with ...
user avatar
  • 11
2 votes
1 answer
170 views

Get the maximum value of all columns 1-3 with common column 4

Given an input txt file: $ cat input.txt -1 23 34 cheese 34 20 15 cheese -4 -4 20 tomato 13 -3 14 tomato 4 6 7 tomato -3 -3 -3 potato I can uniqify the data based on column 4, retaining the ...
user avatar
1 vote
2 answers
2k views

bash remove duplicate lines from txt files in folder

I have this command to remove duplicate lines from a file and save the unique lines in a new file uniq telphone.txt output.txt However Im trying to find a command that scans all .txt files in the ...
user avatar
5 votes
1 answer
321 views

cut command fields

I noticed these two different fields behaviors using cut command: bash:~$ var=`cat /proc/cpuinfo | grep 'model name' | uniq | cut -d ' ' -f 3,4,5,6,7,8 ` echo $var outputs Intel(R) Core(TM) i7-...
user avatar