Skip to main content

Questions tagged [uniq]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1 vote
1 answer
125 views

Why is sorted uniq -c command showing duplicates

I am trying to count how many times I use a certain version of a library on my computer. For some reason, uniq -c is outputing duplicates, despite sorting it, and despite the sort order seeming in ...
Albert's user avatar
  • 161
-1 votes
2 answers
107 views

how to de-duplicate block (timestamp+command) from bash history?

I'm working with bash_history file containing blocks with the following format: #unixtimestamp\ncommand\n here's sample of the bash_history file: #1713308636 cat > ./initramfs/init << "...
Yuki San's user avatar
0 votes
3 answers
64 views

Does a command exist that lists all the directories where a word appears in a file or directory name?

When I don't remember where a file or a folder is, I sometime use the locate command (that finds more occurrences, allow more candidates than a find, in my mind. But maybe I'm mistaking). But then ...
Marc Le Bihan's user avatar
11 votes
1 answer
968 views

Use uniq to filter adjacent lines in pipeline

I'm trying to monitor theme changes using this command: dbus-monitor --session "interface='org.freedesktop.portal.Settings', member=SettingChanged" | grep -o "uint32 ." Output ...
Pavel Skipenes's user avatar
1 vote
5 answers
444 views

Find and delete partially duplicate lines

https://www.domain.com/files/G5SPNDOF/AAA-1080p.mp4.html https://www.domain2.com/dl/G5SPNDOF/JHCGTS/AAA-1080p.mp4.html https://www.domain.com/files/ZQWL80BG/AAA-1080p.mp4.html https://www.domain.com/...
Bogdan Nicolae Stoian's user avatar
0 votes
0 answers
26 views

numeric sort with unique option does not show 0! [duplicate]

I have a file with many redundant numbers in each row. Imagine something like the below: echo "10 9 5 6 4 cell 3 2 0 7 0 1" > test When I use sort -un test I get the following output: ...
javadr's user avatar
  • 131
0 votes
2 answers
203 views

tar processing files multiple times with find -newer

I'm trying to use tar(1) to create an archive of files newer than a specific file (fileA). However, when I use find(1) to obtain the list of files to pass to tar, some files are listed multiple times: ...
Vilinkameni's user avatar
  • 1,210
2 votes
1 answer
8k views

Difference between sort -u and uniq -u

I always have been using sort -u to get rid of duplicates until now. But I am having a real doubt about a list generated by a software tool. The question is: is the output of sort -u |wc the same as ...
Yvain's user avatar
  • 228
0 votes
3 answers
75 views

de-duplicate list but group parts of it

I am compiling some access rules from failed logins and after some piping I arrived at this: cat <<INPUT | sort -k 3,3 --unique Deny from 13.42.98.142 # demo Deny from 13.42.98.142 # test Deny ...
Jonas Eberle's user avatar
14 votes
2 answers
4k views

Who killed my sort? or How to efficient count distinct values from a csv column

I'm doing some processing trying to get how many different lines in a file containing 160,353,104 lines. Here is my pipeline and stderr output. $ tail -n+2 2022_place_canvas_history.csv | cut -d, -f2 |...
wviana's user avatar
  • 213
3 votes
5 answers
786 views

find duplicate 1st field and concat its values in single line

I have a file that has entries in key: value format like the below: cat data.txt name: 'tom' tom_age: '31' status_tom_mar: 'yes' school: 'anne' fd_year_anne: '1987' name: 'hmz' hmz_age: '21' ...
Ashar's user avatar
  • 511
-1 votes
1 answer
110 views

Skip line from console if equal than line before, adding count (in realtime)

Using uniq it is possible to filter out sequential duplicate lines. while (true) do echo 1; echo 2; echo 2; echo 1; sleep 1; done | uniq becomes: 1 2 1 Is there a way to have duplicated sequential ...
Jose Gómez's user avatar
3 votes
2 answers
839 views

Output lines with the same md5 sum

I have a script like this find path -type f -exec md5sum {} +' It has this conclusion /tmp ❯ find $pwd -type f -exec md5sum {} + \a7c8252355166214d1f6cd47db917226 ./guess.bash ...
Котик Вася's user avatar
1 vote
3 answers
280 views

uniq - how to ignore IP address?

I have the following: cat /var/log/example.log | cut -d ' ' -f 3,4,5,6 | sort -u | uniq The result has several thousand lines but I would like to cut this down to show the actual activity and to do ...
CrahanBackup's user avatar
4 votes
2 answers
2k views

Sort and remove duplicates in place and recursively

I would like to sort all files in all subdirectories in place. I have 65536 files in 256 directories, every file containing one word a line, and each file contains duplicates. What I would like is to ...
James's user avatar
  • 71
6 votes
2 answers
5k views

Why doesn't "uniq --unique" remove all duplicate lines?

Running printf "lol\nlol\nfoo\n\n\n\n\nbar\nlol\nlol\nfoo\nlol\nfoo" | uniq --unique prints foo bar foo lol foo Why is foo printed three times? Shouldn't uniq --unique remove them? Also, ...
hanshenrik's user avatar
1 vote
3 answers
1k views

How to sort uniq values from a text file? [duplicate]

I have a text file as shown below: A f1 B f2 A f3 B f4 B f5 I want to sort it based on first column values and keep it in separate files. Desired output: A.txt: A f1 A f3 B.txt: B f2 B f4 B f5 ...
Jyoti Pal's user avatar
2 votes
4 answers
2k views

How can I remove duplicated tags based on specific attribute name in an XML file?

How can I remove duplicate line based on "groupName" and keep the line with directoryId="1"? <Group id="123" groupName="ABC" lowerGroupName="abc" ...
aolaman1's user avatar
1 vote
4 answers
916 views

How do I clean up a log file to show consecutive repeated lines only once, with a number indicating how many times that line repeated?

Current logs: 18:56:54 Info: Starting 18:56:55 Error: timed out 18:56:56 Error: timed out 18:56:57 Error: timed out 18:56:58 Info: reconnected 18:56:59 Error: timed out Desired output: 18:56:54 Info: ...
Paranoid Altoid's user avatar
3 votes
4 answers
2k views

How to remove leading spaces in multiple lines based on indent in first line?

Processing a text line-by-line removing leading spaces on each line is easy: $ LC_ALL=C git ls-files | sed -nE 's:^.*(\.[^./]+)$:\1:p' \ | sort | uniq -c | sort -snr > lines # create example &...
hakre's user avatar
  • 433
2 votes
2 answers
836 views

Why can't I get a unique list of shells? [duplicate]

Trying to get a unique list of shells on my system. When I run this command: cat /etc/passwd | cut -d ':' -f 7 | uniq I get: /bin/bash /bin/sync /sbin/shutdown /sbin/halt /bin/bash I can't ...
Brian Peterson's user avatar
2 votes
3 answers
1k views

Remove duplicated lines containing specific text

I have a big text file containing a million lines. I would like to find identical lines that match my specific text and leave the first occurrence intact. Any ideas? So the algorithm should roughly be ...
nmzik's user avatar
  • 21
7 votes
2 answers
2k views

Print count number of unique pattern occurences after each occurence

Supposing I have a log file containing thrown Exceptions: ExceptionA loggedFunctionCall ExceptionB ExceptionA loggedFunctionCall ExceptionD ExceptionB loggedFunctionCall ExceptionB I want to count ...
komidawi's user avatar
  • 179
14 votes
3 answers
2k views

What is the point of uniq -u and what does it do? [duplicate]

uniq seems to do something different than uniq -u, even though the description for both is "only unique lines". What's the difference here, what do they do?
user11350058's user avatar
0 votes
1 answer
115 views

Linux uniq: how to uniq the list ignore different remark

Original data (abc.csv): 8|AAAAA_001|0| 8|AAAAA_002|0| 8|AAAAA_003|0| 8|AAAAA_004|0| 8|AAAAA_005|0|AAAAA_005 8|AAAAA_006|0| 9|BBBBB_001|0| 9|BBBBB_002|0| 9|BBBBB_003|0|BBBBB_003 9|BBBBB_004|0| 9|...
wilssssssslam's user avatar
1 vote
2 answers
263 views

How to remove duplicate paragraphs

Given a file with paragraphs (group of lines separated by a blank line). It is guaranteed that the line breaks within equivalent paragraphs are conserved. Is there a way of using uniq to remove the ...
Frankie Y. Liu's user avatar
0 votes
2 answers
687 views

How do I find the duplicate lines in a file, output them with the filename, and do this for an entire directory?

I have few files under directory and need to find duplicate lines in each file and display the file name to identify which file has duplicate. I tried: for i in *.*;do sort $i | uniq -d ; done It ...
Satish Kumar's user avatar
-1 votes
1 answer
38 views

print the count and the line after search based on pattern

I have lines like this in multiple log files FILE 1: Line 203:Caused by :arithmatic overflow exception Line 204:at com.abc.java FILE 2: Line 105:Caused by : ???String out of bound exception Line ...
Sathish Kumar's user avatar
3 votes
3 answers
738 views

Sort -u (or sort | uniq) on a couple of columns but retain the info in a third column, and append it to the retained row?

I'm trying to take a table like this: a b hello a b goodbye g g test a c I say to collapse the table down to one row per unique entry on column 1 and 2, but without ...
Jon's user avatar
  • 33
-2 votes
1 answer
384 views

To count the number of occurrences of first field in text file [closed]

i require to count the number of occurrences of first field in txt file and to print output file as two column file having first field of input file & no. of occurrences of fist field in input ...
SBM Anand's user avatar
0 votes
1 answer
94 views

How to use a series of commands on vim

Hi guys in the terminal I use the command cat file.txt | sort | uniq -c . First I would like to use that command on vim and save the output to another file without the output being print when I run ...
Mario's user avatar
  • 5
15 votes
8 answers
4k views

How can I find the most frequent word in a .csv file, ignoring duplicates on each line?

I need to find the 10 most frequent words in a .csv file. The file is structured so that each line contains comma-separated words. If the same word is repeated more than once in the same line, it ...
ginopino's user avatar
  • 380
0 votes
2 answers
803 views

uniq -i is does not ignore case-sensitive in non-Ascii characters

I have a txt: a.txt A a B b Ş ş I try this command and I get false output: $ uniq -ic a.txt 2 A 2 B 1 Ş 1 ş How can I solve the non-ascii character problem ...
rojen's user avatar
  • 184
2 votes
2 answers
369 views

How can I remove blocks of duplicated lines with uniq?

Is there anyway to use uniq (or similar) to filter/remove sets of repeating lines from log type output? I am debugging an MPI code where multiple processors often print the same exact output. uniq ...
rheo's user avatar
  • 31
0 votes
4 answers
399 views

How do I print lines for the first appearance of a unique value in a 2-column file?

I have a small snippet of a file I'm working with: ENSDARG00000032737 ENSDARP00000120731 ENSDARG00000032737 ENSDARP00000049290 ENSDARG00000061051 ENSDARP00000081062 ENSDARG00000061051 ...
gpreising's user avatar
0 votes
1 answer
159 views

Bash command to count lines with matching substrings in two different positions

I'm trying to get some trivial statistics on some debug output. Each debug line is of the form(class name)(delimiter 1)(object ID)(delimiter 2)(method name)(delimiter 3)(log message) I want to get a ...
ShadSterling's user avatar
0 votes
2 answers
2k views

Remove all duplicate files in subdirectories in main directory

I am new to Linux and was wondering how I would go about deleting all duplicates with same file name but only keep one original? For example I have the following: dir_new_001 = rdt.txt dir_new_002 = ...
n00bl1nux's user avatar
2 votes
3 answers
7k views

finding all non-unique lines in a file

I'm trying to use uniq to find all non-unique lines in a file. By non-unique, I mean any line that I have already seen on the previous line. I thought that the "-D" option would do this: -D ...
Michael's user avatar
  • 544
0 votes
2 answers
714 views

Sort command changes string output

I stumbled on this anomaly regarding the sort command and I can't find an explamation to why it happens - running the following command: cat ~/.bash_history | uniq -c | sort -h -r | head Should ...
Danus's user avatar
  • 1
0 votes
3 answers
384 views

Delete line if other line starts with current line (column wise)

I've got a file input.txt in the following tab delimited format: aaaa bbbb aaaa bbbb c aaaa bbbb c dd aaaa bbbb cc aaaa bbbb x aaaa bbbb xx dddd eeee dddd ...
GioWay's user avatar
  • 27
-2 votes
3 answers
548 views

How to count the number of occurrences of all strings in the file while getting the average of each string

I am trying to count the number of occurrences without specifying a specific string, just using the output of the cut command to print the number of each repeated string using grep. I then want to get ...
fgdark's user avatar
  • 23
1 vote
2 answers
2k views

Bash - sort and uniq on array

This is a similar question to this one I want to do the word count but this time, using an array. For example, I have the following IPs inside a bash array called IPS. IPS=("1.1.1.1" "5.5.5.5" "3.3....
Geiser's user avatar
  • 121
0 votes
3 answers
1k views

change format of uniq -c command

I want to get the unique words and its count from a file. When I run the bellow command, sort words.txt | uniq -c 2 america 4 and 1 england 1 file 1 for 1 place But I want the ...
D_Q's user avatar
  • 143
1 vote
2 answers
4k views

Difference between cat file.txt | sort -u and cat file.txt | uniq

I want to know the difference between sort -u and uniq. I tried below command to get the count for the unique strings in a file , sort -u and uniq provides the same output from my knowledge. Then why ...
Joel Deleep's user avatar
2 votes
3 answers
283 views

Remove duplicate lines from file that are in a different order

My file is like this: alice, bob bob, cat cat, dennis cat, bob dennis, alice I want to remove lines where same words have been repeated in reverse order. In this example, bob, cat and cat, bob are ...
user avatar
3 votes
2 answers
555 views

Remove duplicate values that aren't on identical lines

So I have a set of text that contains both the file names and an associated number. Currently it looks like this: RR0.out -1752.142111 RR1.out -1752.141887 RR2.out -1752.142111 RR3.out -...
Pyrodancer123's user avatar
2 votes
2 answers
6k views

How do I count the occurrences in a list, and then sort by descending occurrence?

I have a list with 1601 lines, where on each line, I am interested in counting the # of times different values appear for fields 5. Here's what I have so far: awk '{print$5}' FILE | sort | uniq -c ...
ZakS's user avatar
  • 315
-1 votes
2 answers
343 views

How can i remove lines from a file when a string appears on multiple lines?

I have a file that has 2 columns like the following: apple pear banana pizza spoon fork pizza plate sausage egg If a word appears on multiple lines i want to delete all lines that the repeating word ...
jonny b's user avatar
  • 83
5 votes
1 answer
212 views

Why does uniq drop korean string?

I have a CSV file containing 220k names that I want to batch import into a graph database. The batch importer imposes the restriction that each node must occur only once so I want to remove duplicates ...
oschlueter's user avatar
0 votes
2 answers
2k views

Show line numbers of a file after "cut -d ";" -f3 |uniq -d"

I am trying to make a simple command that can show me the duplicate data from one specific column and also give me the original line number. Example of file: JENNIE;30;DOCTOR;F SARA;26;POLICE;F ...
user350193's user avatar