Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [uniq]

The tag has no usage guidance.

0
votes
1answer
12 views

delete rows with duplications in first column in bash

I have a tab delimited .txt file and want to delete rows that have duplications in the first column. If this can be done without sort, that would be amazing, but I've kind of come to terms with that I ...
0
votes
2answers
47 views

How do I get a shell script to remove duplicates in a text file, based on the 11th 21st columns?

How do I get a shell script to remove duplicates in a text file, based on the 11th 21st columns? Sample file: Header:0000000000000001457854500000 XP 12345678912yeyeyeyeeye 0000003 XP ...
0
votes
4answers
71 views

remove duplicate and replace it with empty space

I have a table as of below fruits shopname Apple x1 orange x1 banana x2 Apple x3 orange x2 banana x3 I want to group all rows based on column 1 and replace the duplicates with ...
2
votes
1answer
77 views

Get the maximum value of all columns 1-3 with common column 4

Given an input txt file: $ cat input.txt -1 23 34 cheese 34 20 15 cheese -4 -4 20 tomato 13 -3 14 tomato 4 6 7 tomato -3 -3 -3 potato I can uniqify the data based on column 4, retaining the ...
1
vote
2answers
68 views

bash remove duplicate lines from txt files in folder

I have this command to remove duplicate lines from a file and save the unique lines in a new file uniq telphone.txt output.txt However Im trying to find a command that scans all .txt files in the ...
4
votes
1answer
176 views

cut command fields

I noticed these two different fields behaviors using cut command: bash:~$ var=`cat /proc/cpuinfo | grep 'model name' | uniq | cut -d ' ' -f 3,4,5,6,7,8 ` echo $var outputs Intel(R) Core(TM) i7-...
0
votes
5answers
131 views

get unique value for each line with unix command

I have a list as below: 1 2 5 2 1 5 5 3 1 5 5 5 5 2 2 2 2 2 4 3 I would like to sort each line and obtain the unique values as below, something similar to sort | uniq : 1 2 5 1 3 5 1 5 2 5 2 3 4 I ...
3
votes
3answers
186 views

Using uniq on unicode text

I want to remove duplicate lines from a file with words of Syriac script. The source file has 3 lines, 1st and 3rd are identical. $ cat file.txt ܐܒܘܢ ܢܗܘܐ ܐܒܘܢ When I use sort and uniq, the result ...
-1
votes
2answers
50 views

print lines with unique specified fields

How to print lines based on having unique fields 2, 3 and 4? fields number 2 and 3 are numbers and field three is alphabet with total 2500 rows in a file. Input: 10,11,12,A,3 10,11,12,A,4 10,11,12,...
1
vote
2answers
303 views

What does `-dD` mean to `uniq`?

From https://unix.stackexchange.com/a/277707/674 find . ! -empty -type f -exec md5sum {} + | sort | uniq -w32 -dD can find duplicate files under the current directory. What does -dD mean to uniq? I ...
-4
votes
2answers
101 views

bash unique line id

Input: file with sorted lines Output: file with (first or any) unique line ids from the input file Example Input abbylove abbylove abbylove abbylover abbylover abbylovesyou abbylovesyou abbylsmith ...
2
votes
4answers
175 views

count number of occurences of substring and keep highest score

I have a list of string like: StringA 45 StrinB 98 StringA 35 StringA 83 StrinB 78 StringC 65 StrinB 98 and I want to filter out duplicates, print the number of occurences (of the substring ...
0
votes
4answers
35 views

Getting Mismatched Entries In 2 Files [closed]

Problem..{ I've users with home dirs in /home, so I can get a list of users matching my criteria with following command. STEP 1: Fetching A List Of Users du -h --max-depth=1 /home 2>/dev/null | ...
1
vote
2answers
196 views

Identify unique records on CSV based on specific columns [duplicate]

If the input is foo,bar,baz bar,baz,qux qux,quux,baz bar,foo,qux waldo,fred,garply the output should be foo,bar,baz bar,baz,qux waldo,fred,garply As you can see, records are deduplicated based on ...
1
vote
1answer
96 views

How do you count the first column generated from uniq -c

I have a file that lists 5 lines of random words "See spot" "See pot run", etc each on a new line. I was able to create code that counted the number of times each word appears in the file and sorted ...
6
votes
2answers
1k views

Keeping unique rows based on information from 2 of three columns

Suppose you have a file like this: NW_006521251.1 428 84134 NW_006521251.1 511 84135 NW_006521038.1 202 84155 NW_006521038.1 1743 84153 NW_006521038.1 1743 84154 NW_006520495.1 198 84159 ...
9
votes
5answers
1k views

Remove adjacent duplicate lines while keeping the order

I have a file with one column with names that repeat a number of times each. I want to condense each repeat into one, while keeping any other repeats of the same name that are not adjacent to other ...
4
votes
3answers
80 views

Script optimisation to find duplicates filename in huge CSV

I have several CSV file from 1MB to 6GB generated by inotify script with a list of events formatted as is: timestamp;fullpath;event;size. Those files are formatted like this : timestamp;fullpath;...
0
votes
2answers
40 views

I can't figure out how to cut this file and find unique words of a particular section

So there's an access log entry file named access_log and I'm supposed to find all of the unique files that were accessed on the web server. access_log is formatted like this, this is just an excerpt: ...
-1
votes
3answers
450 views

bash script: change output uniq -c

I have to change the output of uniq -c: (example) 92 root 80 user in root 92 user 80 how can I do without using awk force?
1
vote
1answer
683 views

Extracting “count value” after using “uniq -c”

I'm playing with bash pipelines using a bunch of NBA data. I'm trying to count the frequency of players on the Cleveland Cavaliers, and right now my command looks something like cut -d "," -f 2 ...
1
vote
1answer
65 views

How to get lines from unique columns?

I have a single ascii file (Report.dat) with these lines: a ./L1/file2.txt c ./L1/file1.txt b ./L2/file1.txt a ./L2/file2.txt and I need to get just the lines where the first column is unique, ...
0
votes
1answer
117 views

What bad thing do “sort” and “uniq” do to the authorized_keys file?

I tried something that seemed to me like it should have worked. I had a bunch of public ssh keys I wanted to import, so I did this. cat public_ssh_keys .ssh/authorized_keys | sort | uniq > AK mv ...
1
vote
3answers
3k views

sort Find how many distinct values are among the certain column

For example, this is the list.txt Joe 3 Jack 1 Ulysses 6 Fox 2 Cassidy 1 Jones 6 Kevin 7 Then the output should be 5 because there are 5 different values in the 2rd column. How should I finish ...
0
votes
2answers
53 views

Why doesn't this process end?

wc -l | sort -t":" -k4,4n | uniq password_demo.txt prints root:x:0:0:root:/root:/bin/bash daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin bin:x:2:2:bin:/bin:/usr/sbin/nologin sys:x:3:3:sys:/dev:/usr/...
-4
votes
1answer
103 views

A lot of duplicates…no fdupes,i want to make a script

No fdup please..I want to make a script. I have a lot of file duplicates,there are more than 200 I made ( is under construction ) a bash script which make md5sum on every file,then with uniq i put ...
-1
votes
1answer
345 views

How to get unique occurence of words from a vey large file?

I have been asked to write a word frequency analysis program using the unix/ shell-scripts with the following requirements: Input is a text file with one word per line Input words are drawn from ...
1
vote
3answers
106 views

remove lines having the same value in a given column

I have the input file (sorted by column 2 with -t,): TOP,25424242,T0137,0.08,0.06,0.02,24 TOP,25424242,T0138,0.07,0.06,0.01,24 TOP,17236110,T0138,9.65,9.37,0.28,89 TOP,23525255,T0137,0.40,0.30,0.11,...
-1
votes
2answers
212 views

Count uniq instances of blocks of 2 lines [closed]

Given input: 144.252.36.69 afrloop=32235330165603 144.252.36.69 afrloop=32235330165603 144.252.36.69 afrloop=32235330165603 222.252.36.69 afrloop=31135330165603 222.252.36.69 afrloop=31135330165603 ...
1
vote
1answer
255 views

assign uniq identifier or serial number to duplicate or uinq values based on any column

i have input file like this which have duplicates values, i want to give uniq identifier for each uniq value based on first column (col1) mins duplicate rows have same identifier before decimal point ...
1
vote
1answer
98 views

Sort based on different fields and values

I have the following data: AD12 Ar.1 352 AD12 Ar.7 495 AD12 Ar.8 491 AD12 Ar.6 495 AD12 Ar.5 495 AD12 Ar.9 491 AD13 Ar2.1 ...
3
votes
1answer
983 views

Less expensive version of `sort -n | uniq -c | sort -n`

I have an unsorted list of IPs that I need to count and sort by occurrences. I use sort -n | uniq -c | sort -n and that works well, but I'd like something less expensive... surely awk can do this? ...
2
votes
1answer
131 views

Check target file for duplicate entry before copy data from source file [duplicate]

I am trying to copy lines from source.txt to target.txt. I would like this bash script to check each line on target.txt if there is duplicate entry before copy. source.txt contains: a$$a$$a b**b**...
2
votes
2answers
11k views

Common lines between two files [duplicate]

I have the following code that I run on my Terminal. LC_ALL=C && grep -F -f genename2.txt hg38.hgnc.bed > hg38.hgnc.goi.bed This doesn't give me the common lines between the two files. ...
1
vote
4answers
550 views

Keeping lines repeated a set number of times [duplicate]

I have a file with multiple columns and have identified lines where values in column 3 have been duplicated. Example Input: A B C 1 2 APPLE 3 4 PEAR 9 3 LEMON 8 3 ORANGE 8 2 APPLE 3 4 APPLE 9 3 ...
3
votes
1answer
9k views

Difference between using `sort -u` and `sort | uniq -u`

I came into a level while exploring the bash that consisted of finding the line of text that occurs only once in a certain file. Why is the output of the sort -u file command different from the ...
3
votes
3answers
97 views

How to remove a specific, duplicated line within a file?

I'm looking for a way to remove one specific line from a bunch of files, but only if it occurs more than once in that file. Other lines should be kept, even if they are duplicates. For example, a ...
2
votes
3answers
1k views

print the duplicate lines only on fields 1,2 from csv file

by the following command we can print the duplicate lines from file uniq -d string file.txt but how we can do it on csv file ? we need to print the duplicate lines only on fields 1,2 from csv file -...
1
vote
2answers
167 views

Using Uniq -c with a regular expression or counting the number of lines removed

I have a tab delimited file with information in it about genetic material. Some information is cut into a smaller tab file with some of the columns extracted, and uniq is used to ensure there are no ...
6
votes
4answers
411 views

How to find the number of unordered pairs from a list

I want to count the number of unordered pairs from a list. In other words, I have a list: ab ba ac bc and I want to display: 2 ab 1 ac 1 bc
0
votes
1answer
72 views

Count lines preserving header

My usual idiom for counting the number of occurrences of a given line in some text input is … | sort | uniq -c | sort -n with the last sort -n optionally used to sort lines by occurrence count. ...
4
votes
1answer
198 views

Why does uniq think あい and いあ are the same?

I'm trying to find duplicates in files, by using: sort myfile | uniq -d I noticed that uniq seems to dislike Japanese characters for some reason. For example, if I have a file: あい いあ Then sort ...
7
votes
4answers
2k views

Efficiently merge / sort / unique large number of text files

I am trying a naive: $ cat * | sort -u > /tmp/bla.txt which fails with: -bash: /bin/cat: Argument list too long So in order to avoid a silly solution like (creates an enormous temporary file): ...
2
votes
3answers
173 views

Using numbers in file A to get unique ID from file B based on the order specified by file A

I want to use the numbers of file whitelist.txt to get a unique identification number from file list.txt based on the order specified by file whitelist.txt. Example: $ cat whitelist.txt: 2 5 7 ...
0
votes
1answer
124 views

Why does uniq utility give me unexpected results? [closed]

I'm creating a shellscript that will print out various filetypes we have in our directory. It pretty much works, however, for some odd reason when I try to use uniq on my output, it doesn't work. This ...
3
votes
3answers
383 views

awk combine two big files and remove duplicated lines [closed]

I have two files: A.txt - about 90GB B.txt - about 80GB I want to combine two files and remove duplicated lines. How do I do this? If commands other than awk are better for this job, ...
1
vote
2answers
750 views

Count unique lines only to a set pattern

How can I count the unique log lines in a text file only until the first "-" and print the line with the count org.springframework. - initialization started org.springframework. - initialization ...
2
votes
1answer
121 views

Keep lines containing only 3 fields

I have a .txt file in this format: USER:EMAIL:PHONE but some lines only have USER:EMAIL I want to extract only the lines containing 3 fields, example USER:EMAIL:PHONE
2
votes
1answer
89 views

Command modification

I'm using the following command to get the highest number of requests per second in a log file and it works well. grep "2017-02-22" "LogFile.log" | cut -c1-20 | uniq -c | sort -n | tail -n1 Now I ...
4
votes
2answers
627 views

Remove duplicate lines with a twist [duplicate]

Okay so I want remove duplicate lines but it's a bit more complicated than that.. I have a file named users.txt, example of file is: users:email@email.com users1:email@email.com Now due to a bug ...