Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [uniq]

The tag has no usage guidance.

1
vote
2answers
31 views

bash remove duplicate lines from txt files in folder

I have this command to remove duplicate lines from a file and save the unique lines in a new file uniq telphone.txt output.txt However Im trying to find a command that scans all .txt files in the ...
4
votes
1answer
154 views

cut command fields

I noticed these two different fields behaviors using cut command: bash:~$ var=`cat /proc/cpuinfo | grep 'model name' | uniq | cut -d ' ' -f 3,4,5,6,7,8 ` echo $var outputs Intel(R) Core(TM) i7-...
0
votes
5answers
98 views

get unique value for each line with unix command

I have a list as below: 1 2 5 2 1 5 5 3 1 5 5 5 5 2 2 2 2 2 4 3 I would like to sort each line and obtain the unique values as below, something similar to sort | uniq : 1 2 5 1 3 5 1 5 2 5 2 3 4 I ...
3
votes
3answers
130 views

Using uniq on unicode text

I want to remove duplicate lines from a file with words of Syriac script. The source file has 3 lines, 1st and 3rd are identical. $ cat file.txt ܐܒܘܢ ܢܗܘܐ ܐܒܘܢ When I use sort and uniq, the result ...
-1
votes
2answers
47 views

print lines with unique specified fields

How to print lines based on having unique fields 2, 3 and 4? fields number 2 and 3 are numbers and field three is alphabet with total 2500 rows in a file. Input: 10,11,12,A,3 10,11,12,A,4 10,11,12,...
1
vote
2answers
292 views

What does `-dD` mean to `uniq`?

From https://unix.stackexchange.com/a/277707/674 find . ! -empty -type f -exec md5sum {} + | sort | uniq -w32 -dD can find duplicate files under the current directory. What does -dD mean to uniq? I ...
-4
votes
2answers
89 views

bash unique line id

Input: file with sorted lines Output: file with (first or any) unique line ids from the input file Example Input abbylove abbylove abbylove abbylover abbylover abbylovesyou abbylovesyou abbylsmith ...
2
votes
4answers
171 views

count number of occurences of substring and keep highest score

I have a list of string like: StringA 45 StrinB 98 StringA 35 StringA 83 StrinB 78 StringC 65 StrinB 98 and I want to filter out duplicates, print the number of occurences (of the substring ...
0
votes
4answers
32 views

Getting Mismatched Entries In 2 Files [closed]

Problem..{ I've users with home dirs in /home, so I can get a list of users matching my criteria with following command. STEP 1: Fetching A List Of Users du -h --max-depth=1 /home 2>/dev/null | ...
1
vote
2answers
129 views

Identify unique records on CSV based on specific columns [duplicate]

If the input is foo,bar,baz bar,baz,qux qux,quux,baz bar,foo,qux waldo,fred,garply the output should be foo,bar,baz bar,baz,qux waldo,fred,garply As you can see, records are deduplicated based on ...
1
vote
1answer
89 views

How do you count the first column generated from uniq -c

I have a file that lists 5 lines of random words "See spot" "See pot run", etc each on a new line. I was able to create code that counted the number of times each word appears in the file and sorted ...
6
votes
2answers
891 views

Keeping unique rows based on information from 2 of three columns

Suppose you have a file like this: NW_006521251.1 428 84134 NW_006521251.1 511 84135 NW_006521038.1 202 84155 NW_006521038.1 1743 84153 NW_006521038.1 1743 84154 NW_006520495.1 198 84159 ...
9
votes
5answers
1k views

Remove adjacent duplicate lines while keeping the order

I have a file with one column with names that repeat a number of times each. I want to condense each repeat into one, while keeping any other repeats of the same name that are not adjacent to other ...
4
votes
3answers
75 views

Script optimisation to find duplicates filename in huge CSV

I have several CSV file from 1MB to 6GB generated by inotify script with a list of events formatted as is: timestamp;fullpath;event;size. Those files are formatted like this : timestamp;fullpath;...
0
votes
2answers
35 views

I can't figure out how to cut this file and find unique words of a particular section

So there's an access log entry file named access_log and I'm supposed to find all of the unique files that were accessed on the web server. access_log is formatted like this, this is just an excerpt: ...
-1
votes
3answers
418 views

bash script: change output uniq -c

I have to change the output of uniq -c: (example) 92 root 80 user in root 92 user 80 how can I do without using awk force?
1
vote
1answer
491 views

Extracting “count value” after using “uniq -c”

I'm playing with bash pipelines using a bunch of NBA data. I'm trying to count the frequency of players on the Cleveland Cavaliers, and right now my command looks something like cut -d "," -f 2 ...
1
vote
1answer
57 views

How to get lines from unique columns?

I have a single ascii file (Report.dat) with these lines: a ./L1/file2.txt c ./L1/file1.txt b ./L2/file1.txt a ./L2/file2.txt and I need to get just the lines where the first column is unique, ...
0
votes
1answer
106 views

What bad thing do “sort” and “uniq” do to the authorized_keys file?

I tried something that seemed to me like it should have worked. I had a bunch of public ssh keys I wanted to import, so I did this. cat public_ssh_keys .ssh/authorized_keys | sort | uniq > AK mv ...
0
votes
3answers
1k views

sort Find how many distinct values are among the certain column

For example, this is the list.txt Joe 3 Jack 1 Ulysses 6 Fox 2 Cassidy 1 Jones 6 Kevin 7 Then the output should be 5 because there are 5 different values in the 2rd column. How should I finish ...
0
votes
2answers
51 views

Why doesn't this process end?

wc -l | sort -t":" -k4,4n | uniq password_demo.txt prints root:x:0:0:root:/root:/bin/bash daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin bin:x:2:2:bin:/bin:/usr/sbin/nologin sys:x:3:3:sys:/dev:/usr/...
-4
votes
1answer
91 views

A lot of duplicates…no fdupes,i want to make a script

No fdup please..I want to make a script. I have a lot of file duplicates,there are more than 200 I made ( is under construction ) a bash script which make md5sum on every file,then with uniq i put ...
-1
votes
1answer
317 views

How to get unique occurence of words from a vey large file?

I have been asked to write a word frequency analysis program using the unix/ shell-scripts with the following requirements: Input is a text file with one word per line Input words are drawn from ...
1
vote
3answers
94 views

remove lines having the same value in a given column

I have the input file (sorted by column 2 with -t,): TOP,25424242,T0137,0.08,0.06,0.02,24 TOP,25424242,T0138,0.07,0.06,0.01,24 TOP,17236110,T0138,9.65,9.37,0.28,89 TOP,23525255,T0137,0.40,0.30,0.11,...
-1
votes
2answers
177 views

Count uniq instances of blocks of 2 lines [closed]

Given input: 144.252.36.69 afrloop=32235330165603 144.252.36.69 afrloop=32235330165603 144.252.36.69 afrloop=32235330165603 222.252.36.69 afrloop=31135330165603 222.252.36.69 afrloop=31135330165603 ...
1
vote
1answer
219 views

assign uniq identifier or serial number to duplicate or uinq values based on any column

i have input file like this which have duplicates values, i want to give uniq identifier for each uniq value based on first column (col1) mins duplicate rows have same identifier before decimal point ...
1
vote
1answer
73 views

Sort based on different fields and values

I have the following data: AD12 Ar.1 352 AD12 Ar.7 495 AD12 Ar.8 491 AD12 Ar.6 495 AD12 Ar.5 495 AD12 Ar.9 491 AD13 Ar2.1 ...
3
votes
1answer
905 views

Less expensive version of `sort -n | uniq -c | sort -n`

I have an unsorted list of IPs that I need to count and sort by occurrences. I use sort -n | uniq -c | sort -n and that works well, but I'd like something less expensive... surely awk can do this? ...
2
votes
1answer
121 views

Check target file for duplicate entry before copy data from source file [duplicate]

I am trying to copy lines from source.txt to target.txt. I would like this bash script to check each line on target.txt if there is duplicate entry before copy. source.txt contains: a$$a$$a b**b**...
2
votes
2answers
8k views

Common lines between two files [duplicate]

I have the following code that I run on my Terminal. LC_ALL=C && grep -F -f genename2.txt hg38.hgnc.bed > hg38.hgnc.goi.bed This doesn't give me the common lines between the two files. ...
1
vote
4answers
484 views

Keeping lines repeated a set number of times [duplicate]

I have a file with multiple columns and have identified lines where values in column 3 have been duplicated. Example Input: A B C 1 2 APPLE 3 4 PEAR 9 3 LEMON 8 3 ORANGE 8 2 APPLE 3 4 APPLE 9 3 ...
2
votes
1answer
8k views

Difference between using `sort -u` and `sort | uniq -u`

I came into a level while exploring the bash that consisted of finding the line of text that occurs only once in a certain file. Why is the output of the sort -u file command different from the ...
3
votes
3answers
92 views

How to remove a specific, duplicated line within a file?

I'm looking for a way to remove one specific line from a bunch of files, but only if it occurs more than once in that file. Other lines should be kept, even if they are duplicates. For example, a ...
2
votes
3answers
930 views

print the duplicate lines only on fields 1,2 from csv file

by the following command we can print the duplicate lines from file uniq -d string file.txt but how we can do it on csv file ? we need to print the duplicate lines only on fields 1,2 from csv file -...
1
vote
2answers
161 views

Using Uniq -c with a regular expression or counting the number of lines removed

I have a tab delimited file with information in it about genetic material. Some information is cut into a smaller tab file with some of the columns extracted, and uniq is used to ensure there are no ...
6
votes
4answers
402 views

How to find the number of unordered pairs from a list

I want to count the number of unordered pairs from a list. In other words, I have a list: ab ba ac bc and I want to display: 2 ab 1 ac 1 bc
0
votes
1answer
72 views

Count lines preserving header

My usual idiom for counting the number of occurrences of a given line in some text input is … | sort | uniq -c | sort -n with the last sort -n optionally used to sort lines by occurrence count. ...
4
votes
1answer
181 views

Why does uniq think あい and いあ are the same?

I'm trying to find duplicates in files, by using: sort myfile | uniq -d I noticed that uniq seems to dislike Japanese characters for some reason. For example, if I have a file: あい いあ Then sort ...
7
votes
4answers
2k views

Efficiently merge / sort / unique large number of text files

I am trying a naive: $ cat * | sort -u > /tmp/bla.txt which fails with: -bash: /bin/cat: Argument list too long So in order to avoid a silly solution like (creates an enormous temporary file): ...
2
votes
3answers
166 views

Using numbers in file A to get unique ID from file B based on the order specified by file A

I want to use the numbers of file whitelist.txt to get a unique identification number from file list.txt based on the order specified by file whitelist.txt. Example: $ cat whitelist.txt: 2 5 7 ...
0
votes
1answer
112 views

Why does uniq utility give me unexpected results? [closed]

I'm creating a shellscript that will print out various filetypes we have in our directory. It pretty much works, however, for some odd reason when I try to use uniq on my output, it doesn't work. This ...
3
votes
3answers
343 views

awk combine two big files and remove duplicated lines [closed]

I have two files: A.txt - about 90GB B.txt - about 80GB I want to combine two files and remove duplicated lines. How do I do this? If commands other than awk are better for this job, ...
1
vote
2answers
609 views

Count unique lines only to a set pattern

How can I count the unique log lines in a text file only until the first "-" and print the line with the count org.springframework. - initialization started org.springframework. - initialization ...
2
votes
1answer
103 views

Keep lines containing only 3 fields

I have a .txt file in this format: USER:EMAIL:PHONE but some lines only have USER:EMAIL I want to extract only the lines containing 3 fields, example USER:EMAIL:PHONE
2
votes
1answer
87 views

Command modification

I'm using the following command to get the highest number of requests per second in a log file and it works well. grep "2017-02-22" "LogFile.log" | cut -c1-20 | uniq -c | sort -n | tail -n1 Now I ...
4
votes
2answers
526 views

Remove duplicate lines with a twist [duplicate]

Okay so I want remove duplicate lines but it's a bit more complicated than that.. I have a file named users.txt, example of file is: users:email@email.com users1:email@email.com Now due to a bug ...
4
votes
1answer
100 views

Generalization of uniq

The uniq(1) command can be viewed as a tiny map/reduce-style program: map consecutive, identical input lines into groups; reduce the list of lines (which are all identical) to a single line. Are ...
6
votes
1answer
246 views

Why can uniq ignore leading characters but not ending characters? [closed]

The POSIX-specified command line tool uniq is designed to "report or filter out repeated lines in a file." Aside from use cases involving the full lines for comparison, there are two options for ...
0
votes
1answer
48 views

Managing duplicated lines across files [closed]

I am generating all the common/duplicate lines between 5 files all within the same directory sort *.txt | uniq -d > output.txt I need output.txt to be merged with File1.txt (so anything that isn'...
0
votes
3answers
80 views

Count unique users in wtmp file

I am struggling to count the unique users in the wtmp file. Here is what I have so far: uniqueUsers=$(last | uniq -c | wc -l) echo "There were '$uniqueUsers' unique users logged on" Any help is ...