Questions tagged [uniq]
The uniq tag has no usage guidance.
173
questions
11
votes
1
answer
933
views
Use uniq to filter adjacent lines in pipeline
I'm trying to monitor theme changes using this command:
dbus-monitor --session "interface='org.freedesktop.portal.Settings', member=SettingChanged" | grep -o "uint32 ."
Output ...
- 235
1
vote
5
answers
356
views
Find and delete partially duplicate lines
https://www.domain.com/files/G5SPNDOF/AAA-1080p.mp4.html
https://www.domain2.com/dl/G5SPNDOF/JHCGTS/AAA-1080p.mp4.html
https://www.domain.com/files/ZQWL80BG/AAA-1080p.mp4.html
https://www.domain.com/...
0
votes
0
answers
26
views
numeric sort with unique option does not show 0! [duplicate]
I have a file with many redundant numbers in each row. Imagine something like the below:
echo "10
9
5
6
4
cell
3
2
0
7
0
1" > test
When I use sort -un test I get the following output:
...
- 131
0
votes
2
answers
126
views
tar processing files multiple times with find -newer
I'm trying to use tar(1) to create an archive of files newer than a specific file (fileA). However, when I use find(1) to obtain the list of files to pass to tar, some files are listed multiple times:
...
- 751
1
vote
1
answer
5k
views
Difference between sort -u and uniq -u
I always have been using sort -u to get rid of duplicates until now.
But I am having a real doubt about a list generated by a software tool.
The question is: is the output of sort -u |wc the same as ...
- 206
0
votes
3
answers
71
views
de-duplicate list but group parts of it
I am compiling some access rules from failed logins and after some piping I arrived at this:
cat <<INPUT | sort -k 3,3 --unique
Deny from 13.42.98.142 # demo
Deny from 13.42.98.142 # test
Deny ...
- 454
14
votes
2
answers
4k
views
Who killed my sort? or How to efficient count distinct values from a csv column
I'm doing some processing trying to get how many different lines in a file containing 160,353,104 lines. Here is my pipeline and stderr output.
$ tail -n+2 2022_place_canvas_history.csv | cut -d, -f2 |...
- 213
3
votes
5
answers
705
views
find duplicate 1st field and concat its values in single line
I have a file that has entries in key: value format like the below:
cat data.txt
name: 'tom'
tom_age: '31'
status_tom_mar: 'yes'
school: 'anne'
fd_year_anne: '1987'
name: 'hmz'
hmz_age: '21'
...
- 459
-1
votes
1
answer
91
views
Skip line from console if equal than line before, adding count (in realtime)
Using uniq it is possible to filter out sequential duplicate lines.
while (true) do echo 1; echo 2; echo 2; echo 1; sleep 1; done | uniq
becomes:
1
2
1
Is there a way to have duplicated sequential ...
- 101
3
votes
2
answers
799
views
Output lines with the same md5 sum
I have a script like this
find path -type f -exec md5sum {} +'
It has this conclusion
/tmp
❯ find $pwd -type f -exec md5sum {} +
\a7c8252355166214d1f6cd47db917226 ./guess.bash
...
- 51
1
vote
3
answers
242
views
uniq - how to ignore IP address?
I have the following:
cat /var/log/example.log | cut -d ' ' -f 3,4,5,6 | sort -u | uniq
The result has several thousand lines but I would like to cut this down to show the actual activity and to do ...
- 11
4
votes
2
answers
1k
views
Sort and remove duplicates in place and recursively
I would like to sort all files in all subdirectories in place. I have 65536 files in 256 directories, every file containing one word a line, and each file contains duplicates.
What I would like is to ...
- 71
5
votes
2
answers
4k
views
Why doesn't "uniq --unique" remove all duplicate lines?
Running
printf "lol\nlol\nfoo\n\n\n\n\nbar\nlol\nlol\nfoo\nlol\nfoo" | uniq --unique
prints
foo
bar
foo
lol
foo
Why is foo printed three times? Shouldn't uniq --unique remove them?
Also, ...
- 585
1
vote
3
answers
931
views
How to sort uniq values from a text file? [duplicate]
I have a text file as shown below:
A f1
B f2
A f3
B f4
B f5
I want to sort it based on first column values and keep it in separate files.
Desired output:
A.txt:
A f1
A f3
B.txt:
B f2
B f4
B f5
...
- 11
2
votes
4
answers
2k
views
How can I remove duplicated tags based on specific attribute name in an XML file?
How can I remove duplicate line based on "groupName" and keep the line with directoryId="1"?
<Group id="123" groupName="ABC" lowerGroupName="abc" ...
- 29
1
vote
4
answers
628
views
How do I clean up a log file to show consecutive repeated lines only once, with a number indicating how many times that line repeated?
Current logs:
18:56:54 Info: Starting
18:56:55 Error: timed out
18:56:56 Error: timed out
18:56:57 Error: timed out
18:56:58 Info: reconnected
18:56:59 Error: timed out
Desired output:
18:56:54 Info: ...
3
votes
4
answers
1k
views
How to remove leading spaces in multiple lines based on indent in first line?
Processing a text line-by-line removing leading spaces on each line is easy:
$ LC_ALL=C git ls-files | sed -nE 's:^.*(\.[^./]+)$:\1:p' \
| sort | uniq -c | sort -snr > lines # create example &...
- 409
2
votes
2
answers
668
views
Why can't I get a unique list of shells? [duplicate]
Trying to get a unique list of shells on my system. When I run this command:
cat /etc/passwd | cut -d ':' -f 7 | uniq
I get:
/bin/bash
/bin/sync
/sbin/shutdown
/sbin/halt
/bin/bash
I can't ...
- 619
2
votes
3
answers
1k
views
Remove duplicated lines containing specific text
I have a big text file containing a million lines.
I would like to find identical lines that match my specific text and leave the first occurrence intact. Any ideas?
So the algorithm should roughly be ...
- 21
5
votes
2
answers
1k
views
Print count number of unique pattern occurences after each occurence
Supposing I have a log file containing thrown Exceptions:
ExceptionA
loggedFunctionCall
ExceptionB
ExceptionA
loggedFunctionCall
ExceptionD
ExceptionB
loggedFunctionCall
ExceptionB
I want to count ...
- 159
14
votes
3
answers
2k
views
What is the point of uniq -u and what does it do? [duplicate]
uniq seems to do something different than uniq -u, even though the description for both is "only unique lines".
What's the difference here, what do they do?
- 159
0
votes
1
answer
101
views
Linux uniq: how to uniq the list ignore different remark
Original data (abc.csv):
8|AAAAA_001|0|
8|AAAAA_002|0|
8|AAAAA_003|0|
8|AAAAA_004|0|
8|AAAAA_005|0|AAAAA_005
8|AAAAA_006|0|
9|BBBBB_001|0|
9|BBBBB_002|0|
9|BBBBB_003|0|BBBBB_003
9|BBBBB_004|0|
9|...
1
vote
2
answers
221
views
How to remove duplicate paragraphs
Given a file with paragraphs (group of lines separated by a blank line).
It is guaranteed that the line breaks within equivalent paragraphs are conserved.
Is there a way of using uniq to remove the ...
0
votes
2
answers
537
views
How do I find the duplicate lines in a file, output them with the filename, and do this for an entire directory?
I have few files under directory and need to find duplicate lines in each file and display the file name to identify which file has duplicate.
I tried: for i in *.*;do sort $i | uniq -d ; done
It ...
-1
votes
1
answer
38
views
print the count and the line after search based on pattern
I have lines like this in multiple log files
FILE 1:
Line 203:Caused by :arithmatic overflow exception
Line 204:at com.abc.java
FILE 2:
Line 105:Caused by : ???String out of bound exception
Line ...
3
votes
3
answers
572
views
Sort -u (or sort | uniq) on a couple of columns but retain the info in a third column, and append it to the retained row?
I'm trying to take a table like this:
a b hello
a b goodbye
g g test
a c I say
to collapse the table down to one row per unique entry on column 1 and 2, but without ...
- 33
-2
votes
1
answer
273
views
To count the number of occurrences of first field in text file [closed]
i require to count the number of occurrences of first field in txt file and to print output file as two column file having first field of input file & no. of occurrences of fist field in input ...
0
votes
1
answer
85
views
How to use a series of commands on vim
Hi guys in the terminal I use the command cat file.txt | sort | uniq -c . First I would like to use that command on vim and save the output to another file without the output being print when I run ...
- 5
15
votes
8
answers
4k
views
How can I find the most frequent word in a .csv file, ignoring duplicates on each line?
I need to find the 10 most frequent words in a .csv file.
The file is structured so that each line contains comma-separated words. If the same word is repeated more than once in the same line, it ...
- 370
0
votes
2
answers
577
views
uniq -i is does not ignore case-sensitive in non-Ascii characters
I have a txt: a.txt
A
a
B
b
Ş
ş
I try this command and I get false output:
$ uniq -ic a.txt
2 A
2 B
1 Ş
1 ş
How can I solve the non-ascii character problem ...
- 184
2
votes
2
answers
337
views
How can I remove blocks of duplicated lines with uniq?
Is there anyway to use uniq (or similar) to filter/remove sets of repeating lines from log type output? I am debugging an MPI code where multiple processors often print the same exact output. uniq ...
- 31
0
votes
4
answers
173
views
How do I print lines for the first appearance of a unique value in a 2-column file?
I have a small snippet of a file I'm working with:
ENSDARG00000032737 ENSDARP00000120731
ENSDARG00000032737 ENSDARP00000049290
ENSDARG00000061051 ENSDARP00000081062
ENSDARG00000061051
...
0
votes
1
answer
132
views
Bash command to count lines with matching substrings in two different positions
I'm trying to get some trivial statistics on some debug output.
Each debug line is of the form(class name)(delimiter 1)(object ID)(delimiter 2)(method name)(delimiter 3)(log message)
I want to get a ...
- 130
0
votes
2
answers
1k
views
Remove all duplicate files in subdirectories in main directory
I am new to Linux and was wondering how I would go about deleting all duplicates with same file name but only keep one original?
For example I have the following:
dir_new_001 = rdt.txt
dir_new_002 = ...
- 33
2
votes
3
answers
5k
views
finding all non-unique lines in a file
I'm trying to use uniq to find all non-unique lines in a file. By non-unique, I mean any line that I have already seen on the previous line. I thought that the "-D" option would do this:
-D ...
- 534
0
votes
2
answers
498
views
Sort command changes string output
I stumbled on this anomaly regarding the sort command and I can't find an explamation to why it happens -
running the following command:
cat ~/.bash_history | uniq -c | sort -h -r | head
Should ...
- 1
0
votes
3
answers
370
views
Delete line if other line starts with current line (column wise)
I've got a file input.txt in the following tab delimited format:
aaaa bbbb
aaaa bbbb c
aaaa bbbb c dd
aaaa bbbb cc
aaaa bbbb x
aaaa bbbb xx
dddd eeee
dddd ...
- 27
-2
votes
3
answers
471
views
How to count the number of occurrences of all strings in the file while getting the average of each string
I am trying to count the number of occurrences without specifying a specific string, just using the output of the cut command to print the number of each repeated string using grep. I then want to get ...
- 23
1
vote
2
answers
2k
views
Bash - sort and uniq on array
This is a similar question to this one
I want to do the word count but this time, using an array.
For example, I have the following IPs inside a bash array called IPS.
IPS=("1.1.1.1" "5.5.5.5" "3.3....
- 121
0
votes
3
answers
910
views
change format of uniq -c command
I want to get the unique words and its count from a file.
When I run the bellow command,
sort words.txt | uniq -c
2 america
4 and
1 england
1 file
1 for
1 place
But I want the ...
- 143
1
vote
2
answers
4k
views
Difference between cat file.txt | sort -u and cat file.txt | uniq
I want to know the difference between sort -u and uniq.
I tried below command to get the count for the unique strings in a file , sort -u and uniq provides the same output from my knowledge. Then why ...
- 239
2
votes
3
answers
276
views
Remove duplicate lines from file that are in a different order
My file is like this:
alice, bob
bob, cat
cat, dennis
cat, bob
dennis, alice
I want to remove lines where same words have been repeated in reverse order. In this example, bob, cat and cat, bob are ...
user199046
3
votes
2
answers
376
views
Remove duplicate values that aren't on identical lines
So I have a set of text that contains both the file names and an associated number. Currently it looks like this:
RR0.out -1752.142111
RR1.out -1752.141887
RR2.out -1752.142111
RR3.out -...
2
votes
2
answers
5k
views
How do I count the occurrences in a list, and then sort by descending occurrence?
I have a list with 1601 lines, where on each line, I am interested in counting the # of times different values appear for fields 5.
Here's what I have so far:
awk '{print$5}' FILE | sort | uniq -c
...
- 285
-1
votes
2
answers
309
views
How can i remove lines from a file when a string appears on multiple lines?
I have a file that has 2 columns like the following:
apple pear
banana pizza
spoon fork
pizza plate
sausage egg
If a word appears on multiple lines i want to delete all lines that the repeating word ...
- 83
5
votes
1
answer
205
views
Why does uniq drop korean string?
I have a CSV file containing 220k names that I want to batch import into a graph database. The batch importer imposes the restriction that each node must occur only once so I want to remove duplicates ...
- 151
0
votes
2
answers
2k
views
Show line numbers of a file after "cut -d ";" -f3 |uniq -d"
I am trying to make a simple command that can show me the duplicate data from one specific column and also give me the original line number.
Example of file:
JENNIE;30;DOCTOR;F
SARA;26;POLICE;F
...
4
votes
2
answers
12k
views
Uniq Descending Sort in a column
In the given data is it possible to uniq sort and print only the top hits against each region?
Given Data
aza1 18
bcn1 16
sat2 12
lcy2 12
fra1 12
aza1 12
bcn1 10
sat2 8
lcy2 9
fra1 13
aza1 21
bcn1 2
...
- 101
2
votes
1
answer
225
views
uniq -c Equivalent for Groups of Lines of Arbitrary Count
I've got a file of ~1-2 million lines that I'm trying to reduce down by counting duplicate groups of lines, preserving order.
uniq -c works okay :
$ perl -E 'say for (("foo") x 4, ("bar") x 4, "baz"...
- 423
0
votes
1
answer
75
views
delete rows with duplications in first column in bash
I have a tab delimited .txt file and want to delete rows that have duplications in the first column. If this can be done without sort, that would be amazing, but I've kind of come to terms with that I ...
- 21