A message from our CEO about the future of Stack Overflow and Stack Exchange. Read now.

Questions tagged [uniq]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
2answers
38 views

Remove all duplicate files in subdirectories in main directory

I am new to Linux and was wondering how I would go about deleting all duplicates with same file name but only keep one original? For example I have the following: dir_new_001 = rdt.txt dir_new_002 = ...
1
vote
3answers
41 views

finding all non-unique lines in a file

I'm trying to use uniq to find all non-unique lines in a file. By non-unique, I mean any line that I have already seen on the previous line. I thought that the "-D" option would do this: -D ...
0
votes
2answers
37 views

Sort command changes string output

I stumbled on this anomaly regarding the sort command and I can't find an explamation to why it happens - running the following command: cat ~/.bash_history | uniq -c | sort -h -r | head Should ...
0
votes
3answers
123 views

Delete line if other line starts with current line (column wise)

I've got a file input.txt in the following tab delimited format: aaaa bbbb aaaa bbbb c aaaa bbbb c dd aaaa bbbb cc aaaa bbbb x aaaa bbbb xx dddd eeee dddd ...
-2
votes
3answers
72 views

How to count the number of occurrences of all strings in the file while getting the average of each string

I am trying to count the number of occurrences without specifying a specific string, just using the output of the cut command to print the number of each repeated string using grep. I then want to get ...
0
votes
2answers
153 views

Bash - sort and uniq on array

This is a similar question to this one I want to do the word count but this time, using an array. For example, I have the following IPs inside a bash array called IPS. IPS=("1.1.1.1" "5.5.5.5" "3.3....
0
votes
3answers
80 views

change format of uniq -c command

I want to get the unique words and its count from a file. When I run the bellow command, sort words.txt | uniq -c 2 america 4 and 1 england 1 file 1 for 1 place But I want the ...
0
votes
2answers
80 views

Difference between cat file.txt | sort -u and cat file.txt | uniq

I want to know the difference between sort -u and uniq. I tried below command to get the count for the unique strings in a file , sort -u and uniq provides the same output from my knowledge. Then why ...
2
votes
3answers
89 views

Remove duplicate lines from file that are in a different order

My file is like this: alice, bob bob, cat cat, dennis cat, bob dennis, alice I want to remove lines where same words have been repeated in reverse order. In this example, bob, cat and cat, bob are ...
3
votes
2answers
49 views

Remove duplicate values that aren't on identical lines

So I have a set of text that contains both the file names and an associated number. Currently it looks like this: RR0.out -1752.142111 RR1.out -1752.141887 RR2.out -1752.142111 RR3.out -...
0
votes
2answers
170 views

How do I count the occurrences in a list, and then sort by descending occurrence?

I have a list with 1601 lines, where on each line, I am interested in counting the # of times different values appear for fields 5. Here's what I have so far: awk '{print$5}' FILE | sort | uniq -c ...
-1
votes
2answers
45 views

How can i remove lines from a file when a string appears on multiple lines?

I have a file that has 2 columns like the following: apple pear banana pizza spoon fork pizza plate sausage egg If a word appears on multiple lines i want to delete all lines that the repeating word ...
4
votes
1answer
70 views

Why does uniq drop korean string?

I have a CSV file containing 220k names that I want to batch import into a graph database. The batch importer imposes the restriction that each node must occur only once so I want to remove duplicates ...
0
votes
2answers
242 views

Show line numbers of a file after “cut -d ”;“ -f3 |uniq -d”

I am trying to make a simple command that can show me the duplicate data from one specific column and also give me the original line number. Example of file: JENNIE;30;DOCTOR;F SARA;26;POLICE;F ...
0
votes
2answers
2k views

Uniq Descending Sort in a column

In the given data is it possible to uniq sort and print only the top hits against each region? Given Data aza1 18 bcn1 16 sat2 12 lcy2 12 fra1 12 aza1 12 bcn1 10 sat2 8 lcy2 9 fra1 13 aza1 21 bcn1 2 ...
2
votes
1answer
75 views

uniq -c Equivalent for Groups of Lines of Arbitrary Count

I've got a file of ~1-2 million lines that I'm trying to reduce down by counting duplicate groups of lines, preserving order. uniq -c works okay : $ perl -E 'say for (("foo") x 4, ("bar") x 4, "baz"...
0
votes
1answer
21 views

delete rows with duplications in first column in bash

I have a tab delimited .txt file and want to delete rows that have duplications in the first column. If this can be done without sort, that would be amazing, but I've kind of come to terms with that I ...
0
votes
2answers
59 views

How do I get a shell script to remove duplicates in a text file, based on the 11th 21st columns?

How do I get a shell script to remove duplicates in a text file, based on the 11th 21st columns? Sample file: Header:0000000000000001457854500000 XP 12345678912yeyeyeyeeye 0000003 XP ...
1
vote
4answers
155 views

remove duplicate and replace it with empty space

I have a table as of below fruits shopname Apple x1 orange x1 banana x2 Apple x3 orange x2 banana x3 I want to group all rows based on column 1 and replace the duplicates with ...
2
votes
1answer
94 views

Get the maximum value of all columns 1-3 with common column 4

Given an input txt file: $ cat input.txt -1 23 34 cheese 34 20 15 cheese -4 -4 20 tomato 13 -3 14 tomato 4 6 7 tomato -3 -3 -3 potato I can uniqify the data based on column 4, retaining the ...
1
vote
2answers
497 views

bash remove duplicate lines from txt files in folder

I have this command to remove duplicate lines from a file and save the unique lines in a new file uniq telphone.txt output.txt However Im trying to find a command that scans all .txt files in the ...
4
votes
1answer
252 views

cut command fields

I noticed these two different fields behaviors using cut command: bash:~$ var=`cat /proc/cpuinfo | grep 'model name' | uniq | cut -d ' ' -f 3,4,5,6,7,8 ` echo $var outputs Intel(R) Core(TM) i7-...
0
votes
5answers
305 views

get unique value for each line with unix command

I have a list as below: 1 2 5 2 1 5 5 3 1 5 5 5 5 2 2 2 2 2 4 3 I would like to sort each line and obtain the unique values as below, something similar to sort | uniq : 1 2 5 1 3 5 1 5 2 5 2 3 4 I ...
4
votes
3answers
597 views

Using uniq on unicode text

I want to remove duplicate lines from a file with words of Syriac script. The source file has 3 lines, 1st and 3rd are identical. $ cat file.txt ܐܒܘܢ ܢܗܘܐ ܐܒܘܢ When I use sort and uniq, the result ...
-1
votes
2answers
69 views

print lines with unique specified fields

How to print lines based on having unique fields 2, 3 and 4? fields number 2 and 3 are numbers and field three is alphabet with total 2500 rows in a file. Input: 10,11,12,A,3 10,11,12,A,4 10,11,12,...
1
vote
2answers
385 views

What does `-dD` mean to `uniq`?

From https://unix.stackexchange.com/a/277707/674 find . ! -empty -type f -exec md5sum {} + | sort | uniq -w32 -dD can find duplicate files under the current directory. What does -dD mean to uniq? I ...
-4
votes
2answers
254 views

bash unique line id

Input: file with sorted lines Output: file with (first or any) unique line ids from the input file Example Input abbylove abbylove abbylove abbylover abbylover abbylovesyou abbylovesyou abbylsmith ...
2
votes
4answers
195 views

count number of occurences of substring and keep highest score

I have a list of string like: StringA 45 StrinB 98 StringA 35 StringA 83 StrinB 78 StringC 65 StrinB 98 and I want to filter out duplicates, print the number of occurences (of the substring ...
0
votes
4answers
51 views

Getting Mismatched Entries In 2 Files [closed]

Problem..{ I've users with home dirs in /home, so I can get a list of users matching my criteria with following command. STEP 1: Fetching A List Of Users du -h --max-depth=1 /home 2>/dev/null | ...
1
vote
2answers
839 views

Identify unique records on CSV based on specific columns [duplicate]

If the input is foo,bar,baz bar,baz,qux qux,quux,baz bar,foo,qux waldo,fred,garply the output should be foo,bar,baz bar,baz,qux waldo,fred,garply As you can see, records are deduplicated based on ...
1
vote
1answer
179 views

How do you count the first column generated from uniq -c

I have a file that lists 5 lines of random words "See spot" "See pot run", etc each on a new line. I was able to create code that counted the number of times each word appears in the file and sorted ...
6
votes
2answers
3k views

Keeping unique rows based on information from 2 of three columns

Suppose you have a file like this: NW_006521251.1 428 84134 NW_006521251.1 511 84135 NW_006521038.1 202 84155 NW_006521038.1 1743 84153 NW_006521038.1 1743 84154 NW_006520495.1 198 84159 ...
11
votes
5answers
2k views

Remove adjacent duplicate lines while keeping the order

I have a file with one column with names that repeat a number of times each. I want to condense each repeat into one, while keeping any other repeats of the same name that are not adjacent to other ...
4
votes
3answers
85 views

Script optimisation to find duplicates filename in huge CSV

I have several CSV file from 1MB to 6GB generated by inotify script with a list of events formatted as is: timestamp;fullpath;event;size. Those files are formatted like this : timestamp;fullpath;...
0
votes
2answers
56 views

I can't figure out how to cut this file and find unique words of a particular section

So there's an access log entry file named access_log and I'm supposed to find all of the unique files that were accessed on the web server. access_log is formatted like this, this is just an excerpt: ...
-1
votes
3answers
577 views

bash script: change output uniq -c

I have to change the output of uniq -c: (example) 92 root 80 user in root 92 user 80 how can I do without using awk force?
1
vote
1answer
2k views

Extracting “count value” after using “uniq -c”

I'm playing with bash pipelines using a bunch of NBA data. I'm trying to count the frequency of players on the Cleveland Cavaliers, and right now my command looks something like cut -d "," -f 2 ...
1
vote
1answer
84 views

How to get lines from unique columns?

I have a single ascii file (Report.dat) with these lines: a ./L1/file2.txt c ./L1/file1.txt b ./L2/file1.txt a ./L2/file2.txt and I need to get just the lines where the first column is unique, ...
0
votes
1answer
179 views

What bad thing do “sort” and “uniq” do to the authorized_keys file?

I tried something that seemed to me like it should have worked. I had a bunch of public ssh keys I wanted to import, so I did this. cat public_ssh_keys .ssh/authorized_keys | sort | uniq > AK mv ...
1
vote
3answers
11k views

sort Find how many distinct values are among the certain column

For example, this is the list.txt Joe 3 Jack 1 Ulysses 6 Fox 2 Cassidy 1 Jones 6 Kevin 7 Then the output should be 5 because there are 5 different values in the 2rd column. How should I finish ...
0
votes
2answers
69 views

Why doesn't this process end?

wc -l | sort -t":" -k4,4n | uniq password_demo.txt prints root:x:0:0:root:/root:/bin/bash daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin bin:x:2:2:bin:/bin:/usr/sbin/nologin sys:x:3:3:sys:/dev:/usr/...
-4
votes
1answer
143 views

A lot of duplicates…no fdupes,i want to make a script

No fdup please..I want to make a script. I have a lot of file duplicates,there are more than 200 I made ( is under construction ) a bash script which make md5sum on every file,then with uniq i put ...
-1
votes
1answer
527 views

How to get unique occurence of words from a vey large file?

I have been asked to write a word frequency analysis program using the unix/ shell-scripts with the following requirements: Input is a text file with one word per line Input words are drawn from ...
1
vote
3answers
258 views

remove lines having the same value in a given column

I have the input file (sorted by column 2 with -t,): TOP,25424242,T0137,0.08,0.06,0.02,24 TOP,25424242,T0138,0.07,0.06,0.01,24 TOP,17236110,T0138,9.65,9.37,0.28,89 TOP,23525255,T0137,0.40,0.30,0.11,...
0
votes
2answers
507 views

Count uniq instances of blocks of 2 lines [closed]

Given input: 144.252.36.69 afrloop=32235330165603 144.252.36.69 afrloop=32235330165603 144.252.36.69 afrloop=32235330165603 222.252.36.69 afrloop=31135330165603 222.252.36.69 afrloop=31135330165603 ...
1
vote
1answer
360 views

assign uniq identifier or serial number to duplicate or uinq values based on any column

i have input file like this which have duplicates values, i want to give uniq identifier for each uniq value based on first column (col1) mins duplicate rows have same identifier before decimal point ...
1
vote
1answer
260 views

Sort based on different fields and values

I have the following data: AD12 Ar.1 352 AD12 Ar.7 495 AD12 Ar.8 491 AD12 Ar.6 495 AD12 Ar.5 495 AD12 Ar.9 491 AD13 Ar2.1 ...
3
votes
1answer
1k views

Less expensive version of `sort -n | uniq -c | sort -n`

I have an unsorted list of IPs that I need to count and sort by occurrences. I use sort -n | uniq -c | sort -n and that works well, but I'd like something less expensive... surely awk can do this? ...
2
votes
1answer
152 views

Check target file for duplicate entry before copy data from source file [duplicate]

I am trying to copy lines from source.txt to target.txt. I would like this bash script to check each line on target.txt if there is duplicate entry before copy. source.txt contains: a$$a$$a b**b**...
10
votes
2answers
30k views

Common lines between two files [duplicate]

I have the following code that I run on my Terminal. LC_ALL=C && grep -F -f genename2.txt hg38.hgnc.bed > hg38.hgnc.goi.bed This doesn't give me the common lines between the two files. ...