Linked Questions

0
votes
1answer
2k views

Delete duplicate entries in a text file [duplicate]

I created a txt file using two requests, one LDAP and one SQL. Results of the two requests are stored in the same txt file. The txt file looks like this : user1@domain.fr user2@domain.fr user3@...
0
votes
1answer
680 views

How can i remove duplicates from the output of tshark output [duplicate]

sudo tshark -i ppp0 'tcp port 80 \ and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)' \ -R'http.request.method == "GET" && http.request.uri contains "/ABC/...
1
vote
1answer
494 views

remove all duplicates from a text file without sort [duplicate]

simply put, I have a file with lines of text that are unknown to me, something like abaa dddd bbbb cccc abaa aaaa abaa the result I'd like to get is dddd bbbb cccc aaaa where all the duplicates ...
-1
votes
1answer
561 views

How to remove duplicate lines from file? [duplicate]

I have file for example 'a' aaa aaa bbb ccc ccc bbb ddd After executing uniq a c i get file 'c' aaa bbb ccc bbb ddd How to delete duplicate bbb lines?
191
votes
4answers
37k views

Why is using a shell loop to process text considered bad practice?

Is using a while loop to process text generally considered bad practice in POSIX shells? As Stéphane Chazelas pointed out, some of the reasons for not using shell loop are conceptual, reliability, ...
37
votes
2answers
14k views

How does awk '!a[$0]++' work?

This one-liner removes duplicate lines from text input without pre-sorting. For example: $ cat >f q w e w r $ awk '!a[$0]++' <f q w e r $ The original code I have found on the internets read:...
8
votes
5answers
3k views

Remove duplicate lines from a file that contains a timestamp

This question/answer has some good solutions for deleting identical lines in a file, but won't work in my case since the otherwise duplicate lines have a timestamp. Is it possible to tell awk to ...
16
votes
1answer
8k views

How to remove duplicate lines in a large multi-GB textfile?

My question is similar to this question but with a couple of different constraints: I have a large \n delimited wordlist -- one word per line. Size of files range from 2GB to as large as 10GB. I ...
3
votes
3answers
499 views

awk combine two big files and remove duplicated lines [closed]

I have two files: A.txt - about 90GB B.txt - about 80GB I want to combine two files and remove duplicated lines. How do I do this? If commands other than awk are better for this job, ...
4
votes
2answers
340 views

Find the top 5 (according to number of packets sent) source IP addresses

I am doing an assignment, I'm asked to answer certain questions based on pcap file that I'm given. One of the question is to find the top 5 (according to number of packets sent) source IP addresses. ...
2
votes
2answers
564 views

Removing duplicates in a large text list

I've searched around the internet and stackexchange for this. Even though there are lots of similar topics, I haven't found a solution yet. So, I have a quite large list (approx. 20GB), which ...
1
vote
2answers
346 views

Append words to wordlist with sort -u avoinding duplicata

I have 2 txt file called one.txt with duplicata: yesterday yesterday today today tomorrow tomorrow and the second txt called two.txt with duplicata: mike mike paul paul tomorrow tomorrow Using the ...
0
votes
1answer
269 views

How to use awk to print nth column and remove duplicates?

I am using awk below to print 8th column and remove duplicates in that very column. awk -F "," '{print $8}' filecsv | awk '!NF || !seen[$0]++' How to do it with just one awk instead running awk ...