Linked Questions
22 questions linked to/from How does awk '!a[$0]++' work?
2
votes
1
answer
2k
views
how awk seen option works [duplicate]
I have recently come across awk seen option. I can see it is removing duplicates in files. But unsure how it works tried google but didnt find any results, need help in clarification for the same.
cat ...
- 51
0
votes
1
answer
81
views
Please explain this awk statement [duplicate]
Yesterday I was googling how to merge two files and came across an awk snippet.
I need a simple merge, so sort -u is not the way to go, but the code below works.
Could some one please explain what ...
- 1
199
votes
10
answers
255k
views
How to remove duplicate lines inside a text file?
A huge (up to 2 GiB) text file of mine contains about 100 exact duplicates of every line in it (useless in my case, as the file is a CSV-like data table).
What I need is to remove all the repetitions ...
- 16.2k
9
votes
2
answers
16k
views
Remove entire row in a file if first column is repeated
I have a file containing two columns and 10 million rows. The first column contains many repeated values, but there is a distinct value in column 2. I want to remove the repeated rows and want to keep ...
- 301
6
votes
4
answers
812
views
How to remove unique strings from a textfile?
Sorry guys I had to edit my example, because I didn't express my query properly.
Let's say I have the .txt file:
Happy sad
Happy sad
Happy sad
Sad happy
Happy sad
Happy sad
Mad sad
Mad happy
Mad happy
...
- 151
4
votes
3
answers
3k
views
Print line if value in column changes
I know this should be an easy one by googling but was not successful. Sorry for that.
I would like to print the first line of groups defined the value in the first column. Delimiter is tab.
Input:
...
- 193
4
votes
2
answers
2k
views
Why doesn't "uniq --unique" remove all duplicate lines?
Running
printf "lol\nlol\nfoo\n\n\n\n\nbar\nlol\nlol\nfoo\nlol\nfoo" | uniq --unique
prints
foo
bar
foo
lol
foo
Why is foo printed three times? Shouldn't uniq --unique remove them?
Also, ...
- 463
5
votes
1
answer
8k
views
In awk, how can I make a boolean value that I can toggle it?
In other programming languages there often has a bool or boolean type. I can create a boolean type variable and use the not operator to toggle it. I can toggle it many times and get a series of true, ...
- 1,626
5
votes
2
answers
2k
views
delete a line if same line exists in the previous line
I want to remove lines which exist already in the previous line from the command line in UNIX. I have the following data in a file.
<xref id="gi_525506931_ref_NP_001266519.1__brain_aromatase"/>...
- 169
2
votes
3
answers
368
views
Removing Duplicates from a CSV based on specified columns
I am working with a CSV data set which looks like the below:
year,manufacturer,brand,series,variation,card_number,card_title,sport,team
2015,Leaf,Trinity,Printing Plates,Magenta,TS-JH2,John Amoth,...
- 123
0
votes
4
answers
2k
views
read phone numbers from file and store them in other file uniquely
I have input text file e.g myfile.txt which is contains data like
WO_ID
------------------------------------------------------------------------
moveover_virus_8493020020_virus.final
...
- 3
0
votes
2
answers
1k
views
Retrieve first occurrence of record, where matching pattern is taken from input
I have a list like this:
2017-12-11 AAOI 40.33
2017-11-15 AAOI 44.3492
2017-12-15 AEIS 70.98
2017-11-15 AEIS 80.137
2017-10-23 AIEQ 25.1601
2017-11-15 AMBA 52.6501
2017-12-05 ...
- 85
1
vote
5
answers
197
views
How can I delete any line containing 2 or more words (which aren't separated by space) from text file? [closed]
How can I delete any line containing 2 or more words (which aren't separated by space) from text file?
The file has also "single versions" of those words.
For example for:
alpha
beta
gama
...
- 47
1
vote
3
answers
296
views
awk operator precedence of `!a[$0]++`
I have an awk expression !a[$0]++ and I want to know the exact evaluation order. From the document of awk, ++ is at higher precedence than !. However, in this example, ++ is in its postfix form (not !+...
- 1,626
3
votes
2
answers
398
views
Concatenate files that overlap, avoiding repetition
Assume we have two text files and we want to combine them into one.
The second file starts with lines that are also in the first file, so it repeats part of it. There is a redundant overlap.
How can I ...
- 15.9k