Linked Questions

0
votes
1answer
66 views

Please explain this awk statement [duplicate]

Yesterday I was googling how to merge two files and came across an awk snippet. I need a simple merge, so sort -u is not the way to go, but the code below works. Could some one please explain what ...
0
votes
0answers
39 views

What is awk '!a[$0]++' doing [duplicate]

I need to understand the "awk '!a[$0]++' " command. I know that this is removing the duplicate lines.
124
votes
9answers
173k views

How to remove duplicate lines inside a text file?

A huge (up to 2 GiB) text file of mine contains about 100 exact duplicates of every line in it (useless in my case, as the file is a CSV-like data table). What I need is to remove all the repetitions ...
7
votes
2answers
14k views

Remove entire row in a file if first column is repeated

I have a file containing two columns and 10 million rows. The first column contains many repeated values, but there is a distinct value in column 2. I want to remove the repeated rows and want to keep ...
2
votes
3answers
2k views

Print line if value in column changes

I know this should be an easy one by googling but was not successful. Sorry for that. I would like to print the first line of groups defined the value in the first column. Delimiter is tab. Input: ...
5
votes
2answers
1k views

delete a line if same line exists in the previous line

I want to remove lines which exist already in the previous line from the command line in UNIX. I have the following data in a file. <xref id="gi_525506931_ref_NP_001266519.1__brain_aromatase"/>...
4
votes
1answer
3k views

In awk, how can I make a boolean value that I can toggle it?

In other programming languages there often has a bool or boolean type. I can create a boolean type variable and use the not operator to toggle it. I can toggle it many times and get a series of true, ...
0
votes
4answers
764 views

read phone numbers from file and store them in other file uniquely

I have input text file e.g myfile.txt which is contains data like WO_ID ------------------------------------------------------------------------ moveover_virus_8493020020_virus.final ...
0
votes
1answer
419 views

How to sort or uniq a live feed

I'm looking to sort and isolate IP from a tcpdump live feed. tcpdump -n -i tun0 "tcp[tcpflags] & (tcp-syn) != 0" | grep -E -o "([0-9]{1,3}[\.]){3}[0-9]{1,3} works just fine but when I try to add ...
0
votes
2answers
185 views

Retrieve first occurrence of record, where matching pattern is taken from input

I have a list like this: 2017-12-11 AAOI 40.33 2017-11-15 AAOI 44.3492 2017-12-15 AEIS 70.98 2017-11-15 AEIS 80.137 2017-10-23 AIEQ 25.1601 2017-11-15 AMBA 52.6501 2017-12-05 ...
3
votes
1answer
70 views

Explaining awk code

Code is being used to remove duplicate entries from histroy.txt file contains history of command line. BEGIN{ if (data[$0]++ == 0) lines[++count] = $0; } END { for(i=1; i<...
1
vote
2answers
64 views

Remove duplicates from particular columns

I have a file in below format, where columns are separated with comma. [1], Value1, UAC, AB [2.2], Check1, BOH D2A D2A BOH, SD [63], name2, MFB MFB, k ... I want ...