The tag has no usage guidance.

learn more… | top users | synonyms

0
votes
1answer
31 views

`uniq` is not realtime when piped

When I run for i in {1..2000}; do sleep 0.1 && echo $(netstat -s | grep -i syns); done | uniq i can see in realtime when expression $(netstat -s | grep -i syns) changes. But when I want to ...
0
votes
1answer
38 views

How to sort or uniq a live feed

I'm looking to sort and isolate IP from a tcpdump live feed. tcpdump -n -i tun0 "tcp[tcpflags] & (tcp-syn) != 0" | grep -E -o "([0-9]{1,3}[\.]){3}[0-9]{1,3} works just fine but when I try to ...
0
votes
1answer
47 views

How can I remove duplicate entries in linux log

I've turned mysql general log on and I've been logging every transaction that occurs for the last week. I also have a cronjob running every 5 minutes doing some mysql transactions which is polluting ...
-5
votes
3answers
76 views

How to write a Shell Script that will count the occurrences of each number in a file?

This question was asked by the interviewer today. Suppose there is a file named "Myfile.txt" and it's contains are as below:(not necessarily in one line) 1 3 4 4 1 5 1 9 8 3 2 1 6 0 0 3 4 5 7 8 9 I ...
3
votes
4answers
150 views

Trying to sort two list of numbers and using uniq to get the intersection

I have a file A and B so I used to the following command... (sort -n A B) | uniq -d which should give me the numbers which occur in both files. 1 2 2 3 4 5 11 11 12 31 These are the numbers I ...
15
votes
12answers
2k views

Delete duplicate lines pairwise?

I encountered this use case today. It seems simple at first glance, but fiddling around with sort, uniq, sed and awk revealed that it's nontrivial. How can I delete all pairs of duplicate lines? In ...
0
votes
1answer
33 views

how to print the count of matched /unmatched patterns and print multiple patterns separated by line

how to print the count of matched /unmatched patterns and print multiple patterns separated by line. Input example (test.log): This 23 line has eight 8888 This 11 line has three 3333 need 12 to ...
1
vote
1answer
52 views

Filter out duplicate lines of tail -f [duplicate]

I often need to tail -f apache access logs for websites to troubleshoot issues- one thing that makes it annoying is that anyone loading a page once may cause 12+ lines to get written to the log, and ...
4
votes
3answers
391 views

Remove nearly duplicate lines

I've got a knotty problem that I can't figure out how to solve. I have a text file containing a few million lines of text. Basically I want to run uniq, but with a twist: If two lines are identical ...
0
votes
1answer
54 views

Monitoring Server Connections - Netstat formatting issue

I have had a few issues with a server recently. So i just wanted to leave a window showing the unique and IP's of connected devices. I have been using: watch -n 5 "netstat -tn 2>/dev/null | grep :...
2
votes
5answers
2k views

How let `sort | uniq -c` separate the number of occurrences by a tab charactor?

uniq -c separate the occurrence number by spaces, which is hard for to cut or awk to separate it out later. 1000_A1\tB1\n ___1_A2\tB2\n I can solve this problem using sed -r 's/^ *([0-9]+)/\1\t/' ...
3
votes
1answer
33 views

Sort log data with bash

I have a log file containing such entries 192.168.1.1;query=/etc/security/limits.conf;date=02.03.16 192.168.1.1;query=/etc/security/limits.conf;date=01.06.15 192.168.1.2;query=/etc/security/limits....
22
votes
5answers
8k views

How can I remove duplicates in my .bash_history, preserving order?

I really enjoying using control+r to recursively search my command history. I've found a few good options I like to use with it: # ignore duplicate commands, ignore commands starting with a space ...
3
votes
4answers
86 views

Find duplicates by column in a file

I have an input file containing file paths and their md5sum, separated by semicolons and sorted by md5 hashes : /media/LaCie/Images/recup2/recup_dir.1/f1295328.jpg;0080ececd3da8533f5d11e449cf73287 /...
0
votes
1answer
75 views

linux compare text file [duplicate]

I have file1.txt and file2.txt, both contain URLs, I want to create a new file with the unique URLS present in file2.txt and not in file1.txt. Example urls file: aspencabinets.com.au iiatca.org ...
0
votes
2answers
28 views

find unique elements restricted by column

The unix command uniq -u returns truly unique elements (as opposed to the default behavior of uniq). For example: echo -e "a\na\nb\nc\nc\nd\ne\ne\ne\nf" | uniq -u b d f How can this command ...
0
votes
1answer
125 views

sort and uniq commands not running as expected when run though cron

Below is a part of script which gives proper output when run manually but gives incorrect output when run using cron: sort < file1.out | uniq -ic |sort -nr> file2.out When run on the command ...
3
votes
1answer
171 views

Unix - count unique IP addresses, sort them by most frequent and also sort them by IP when number of repetitions is same

I have a list of IP addresses in file: 72.204.55.250 72.204.55.250 72.204.55.250 72.204.55.250 72.204.55.250 96.41.51.202 208.115.113.91 178.137.94.166 178.137.94.166 208.115.113....
0
votes
2answers
61 views

Prints duplicate lines based on first column

I want to get lines where first column duplicates (I used number 32 becouse first column has always 32 characters). So far I've figured out: sort | uniq -d -w 32 filename The problem is that I want ...
1
vote
1answer
66 views

Why does this command not sort based on the uniq count?

I have lines in a log similar to: 2015/11/02-07:55:39.735 INFO failed with ERR_AUTHORIZATION_REQUIRED. (10.10.10.11:61618) is not a trusted source. 2015/11/02-07:55:40.515 INFO failed with ...
4
votes
5answers
3k views

How to create an array of unique elements from a string/array in bash?

If I have a string "1 2 3 2 1" - or an array [1,2,3,2,1] - how can I select the unique values, i.e. "1 2 3 2 1" produces "1 2 3" or [1,2,3,2,1] produces [1,2,3] Similar to uniq but uniq seems ...
0
votes
1answer
52 views

How to find number of unique 'applications' in Debian jessie

This is a follow-up query on allpackages.txt.gz has two different file-sizes? $ wc -l allpackages.txt 57725 allpackages.txt I looked at the output and it says 57725 -6 the header files which is ...
2
votes
1answer
68 views

“Count how many different starting letters are in languages names”

So I have this text document with 89 lines. In this document the first column of every line is another language. My task is to count how many different starting letters are used in the document. I ...
2
votes
3answers
188 views

uniq showing duplicate lines

$ grep home american.txt | sort | uniq A home and a country should leave us no more! ...
60
votes
5answers
58k views

What is the difference between “sort -u” and “sort | uniq”?

Everywhere I see someone needing to get a sorted, unique list, they always pipe to sort | uniq. I've never seen any examples where someone uses sort -u instead. Why not? What's the difference, and why ...
2
votes
3answers
60 views

Listed Frequency of Different Strings in a Particular Column

So I'm fairly new to linux, and needed help figuring out how to find how many times a particular string shows up in column 4. This is my data: 25 48656721 48656734 FAM132B ENSCAFT00000019683 4 0.51 ...
-1
votes
2answers
190 views

select portion of file by line interval [duplicate]

I want to practice using head, uniq and cut, for this data here. I know this thread but it focuses too much on cat and external programs. I would like a simpler answer. I want to take interval from ...
5
votes
2answers
1k views

Fastest `uniq` tool in linux

I have large text file (1.5 G), I want to know what the fastest and more reliable tool in linux is. I usualy use: awk '!x[$0]++' file.txt But when I use htop command I see my memory usage is ...
5
votes
4answers
426 views

How to find duplicate lines in a text file, while some may be commented out or have different tokens at the beginning?

I have a text file with lines that are a mixture like this: ### Comments # Comments 86.242.200.81 banana.domain.net # comment 86.242.200.3 orange.domain.net 31.28.225.81 monkey....
4
votes
2answers
350 views

uniq and bash for loop not writing to stdout before stdin closing (for one-line website visitor notification system)

I'm trying to trigger a beep on the PC speaker for every unique visitor of a website. After some brainstorming, it seemed to be possible with one line: for e in `ssh me@mymachine "tail -n 1 -f /var/...
3
votes
2answers
344 views

sort and uniq columns individually in a text file

I have a file as below: D F T E A R T E K A O E E T P J I would like to sort each column and unique the columns individually as below: A A O E D F P J E R T K T I wonder if there is anyone who ...
0
votes
2answers
266 views

Display filenames with grep

I'm looking for a way to display the filename when using this command: cat *.tcp | grep "tcp" | grep "open" | sort | uniq Is there a way to do it?
32
votes
4answers
3k views

How is uniq not unique enough that there is also uniq --unique?

Here are commands on a random file from pastebin: wget -qO - http://pastebin.com/0cSPs9LR | wc -l 350 wget -qO - http://pastebin.com/0cSPs9LR | sort -u | wc -l 287 wget -qO - http://pastebin.com/...
0
votes
1answer
228 views

Sort -u without sorting but with better uniqueness? [duplicate]

I don't want to sort my file, just filter out duplicate lines, maintaining the original ordering. Is there a way to use sort's unique function without it's sort function (something like cat -u would ...
1
vote
2answers
48 views

Print log lines excluding previously-repetitive lines?

If I had a log file looking like this: A A B C C C A B B I would like to output (remove the immediately-successive duplicates): A B C A B How do I do this?
16
votes
2answers
3k views

How does awk '!a[$0]++' work?

This one-liner removes duplicate lines from text input without pre-sorting. For example: $ cat >f q w e w r $ awk '!a[$0]++' <f q w e r $ The original code I have found on the internets ...
2
votes
1answer
55 views

Stateful uniq functionality

I've a program producing large amount of redundant information. I'd like to store only unique lines produced on stdout, but I can't do it in multiple steps, like: ./prog > out cat out | sort | ...
0
votes
1answer
2k views

How to use uniq Unix command and specify input and output file?

I'm confused on the use of the uniq command. Can anyone explain this command and how to use it? How can I use uniq without specifying any option? Any example would be helped and appreciated.
2
votes
2answers
542 views

Count & sum no. of occurrences of each unique line in a file [duplicate]

I think the answer to this problem might be some sort of variant of the uniq function that allows you to count the number of times each line appears in a file: sort file.txt | uniq -c The problem ...
-1
votes
1answer
139 views

sort | uniq not working on different files [closed]

I have no idea why the command sort filename.txt | uniq is not working on this file: http://packetstormsecurity.com/files/32089/koran.gz.html I have also used: awk '!seen[$0]++' koran.txt perl -...
2
votes
1answer
4k views

sort and uniq in awk

I know there are "sort" and "uniq" out there, however, today's question is about how to utilise AWK to do that kind of a job. Say if I have a list of anything really (ips, names, or numbers) and I ...
6
votes
3answers
3k views

sum columns based on matching fields

I have encountered a problem that exceeds my basic unix knowledge and would really appreciate some help. I have a large file in the following format: 2 1019 0 12 2 1019 3 0 2 1021 0 2 ...
1
vote
3answers
193 views

lines with unique checksum

I have a file A of lines of equal length n. I have a program that for each line in A can output a line of length n which its key. The program can either write the sequence of keys for the lines in A ...
1
vote
1answer
374 views

Removing duplicate lines after comparing first pattern and special pattern

I'm looking for code to removing duplicate lines after comparing the first tab-delimited pattern: Example : (↦ = tab) car ↦ ab ↦ 38/43 car ↦ ca ↦ 2/34 house ↦ ca ↦ 34/34 house ↦ d ↦ 2/3 house ↦ ht ↦ ...
1
vote
2answers
184 views

How to match a number in one file to a name in another file?

Okay, I hope this is the right place to ask. I have 2 files. The first file has 4 fields: class CRN #, Instructor ID #, Class Section #, Number of Students. Example: 0002 T00005 006 15 0004 ...
2
votes
3answers
262 views

get list of months using dseq from dateutils

how'd I get a listing of the months in the year (abbreviated) using dseq from dateutils? NOTE simply running: dseq 'jan' 'dec' -i '%b' -f '%b' returns a humongous list of months: piping the ...
1
vote
2answers
171 views

Getting the size of parent folders when using find to search for criteria?

I am hunting for a certain regular expression within the filenames and am seeking to return the parent folders with their sizes in a unique fashion. For example, here is the command that I expected ...
2
votes
1answer
38 views

Why might these values not be uniq'ed together?

I am trying to figure out which IP addresses hit the server most often. I received this surprising output from uniq: $ cat access_log | awk '{print $1}' | uniq -c | sort -bnr | head 26 73.23.5.25 ...
2
votes
2answers
208 views

Removing lines with a single common field

I have a tab separated file that looks like this: 123 some text 123 some different text 334 some other text 341 more text and I want to do two things. One is to order everything numerically ...
7
votes
4answers
391 views

Groupwise uniq command?

I am searching for a command to get from a file in this format: hello 32 hello 67 hi 2 ho 1212 ho 1390 ho 3000 To this format (deduplicate by taking the last row of a "group"): hello ...