Linked Questions

0
votes
1answer
91 views

how awk seen option works [duplicate]

I have recently come across awk seen option. I can see it is removing duplicates in files. But unsure how it works tried google but didnt find any results, need help in clarification for the same. cat ...
0
votes
1answer
73 views

Please explain this awk statement [duplicate]

Yesterday I was googling how to merge two files and came across an awk snippet. I need a simple merge, so sort -u is not the way to go, but the code below works. Could some one please explain what ...
158
votes
10answers
212k views

How to remove duplicate lines inside a text file?

A huge (up to 2 GiB) text file of mine contains about 100 exact duplicates of every line in it (useless in my case, as the file is a CSV-like data table). What I need is to remove all the repetitions ...
7
votes
2answers
15k views

Remove entire row in a file if first column is repeated

I have a file containing two columns and 10 million rows. The first column contains many repeated values, but there is a distinct value in column 2. I want to remove the repeated rows and want to keep ...
6
votes
4answers
672 views

How to remove unique strings from a textfile?

Sorry guys I had to edit my example, because I didn't express my query properly. Let's say I have the .txt file: Happy sad Happy sad Happy sad Sad happy Happy sad Happy sad Mad sad Mad happy Mad happy ...
3
votes
3answers
3k views

Print line if value in column changes

I know this should be an easy one by googling but was not successful. Sorry for that. I would like to print the first line of groups defined the value in the first column. Delimiter is tab. Input: ...
5
votes
1answer
6k views

In awk, how can I make a boolean value that I can toggle it?

In other programming languages there often has a bool or boolean type. I can create a boolean type variable and use the not operator to toggle it. I can toggle it many times and get a series of true, ...
5
votes
2answers
1k views

delete a line if same line exists in the previous line

I want to remove lines which exist already in the previous line from the command line in UNIX. I have the following data in a file. <xref id="gi_525506931_ref_NP_001266519.1__brain_aromatase"/>...
0
votes
4answers
2k views

read phone numbers from file and store them in other file uniquely

I have input text file e.g myfile.txt which is contains data like WO_ID ------------------------------------------------------------------------ moveover_virus_8493020020_virus.final ...
0
votes
2answers
677 views

Retrieve first occurrence of record, where matching pattern is taken from input

I have a list like this: 2017-12-11 AAOI 40.33 2017-11-15 AAOI 44.3492 2017-12-15 AEIS 70.98 2017-11-15 AEIS 80.137 2017-10-23 AIEQ 25.1601 2017-11-15 AMBA 52.6501 2017-12-05 ...
1
vote
5answers
139 views

How can I delete any line containing 2 or more words (which aren't separated by space) from text file? [closed]

How can I delete any line containing 2 or more words (which aren't separated by space) from text file? The file has also "single versions" of those words. For example for: alpha beta gama ...
1
vote
3answers
199 views

awk operator precedence of `!a[$0]++`

I have an awk expression !a[$0]++ and I want to know the exact evaluation order. From the document of awk, ++ is at higher precedence than !. However, in this example, ++ is in its postfix form (not !+...
0
votes
1answer
522 views

How to sort or uniq a live feed

I'm looking to sort and isolate IP from a tcpdump live feed. tcpdump -n -i tun0 "tcp[tcpflags] & (tcp-syn) != 0" | grep -E -o "([0-9]{1,3}[\.]){3}[0-9]{1,3} works just fine but when I try to add ...
3
votes
1answer
77 views

Explaining awk code

Code is being used to remove duplicate entries from histroy.txt file contains history of command line. BEGIN{ if (data[$0]++ == 0) lines[++count] = $0; } END { for(i=1; i<...
1
vote
2answers
303 views

Remove duplicates from particular columns

I have a file in below format, where columns are separated with comma. [1], Value1, UAC, AB [2.2], Check1, BOH D2A D2A BOH, SD [63], name2, MFB MFB, k ... I want ...

15 30 50 per page