1

I have a file named basefile.txt with huge number of lines. The records in the file are logically grouped based on their sets.

E.g.:

"GRP_START","LINE1"........
"A"
"B--BOOM"
"C"
"GRP_START","LINE1"........
"A"
"B--DOOM"
"C"
"D"
"E"
"F"
"G"
"GRP_START","LINE1"........
"E"
"F"
"G"
"C--MOOM"
"GRP_START","LINE1"........

What I want to do is, I have those these search strings in a separate file named keywords.txt which contains below data:

BOOM
DOOM
MOOM

I need a command which will read the text one by one from keywords.txt file and search that text in the basefile.txt and write certain records into a finalfile.txt

Those certain records are: I need the Sets where the search text is falling.

E.g: In the above files.

the command should search for BOOM and return the below lines

"GRP_START","LINE1"........
"A"
"B--BOOM"
"C"

the command should search for DOOM and return the below lines

"GRP_START","LINE1"........
"A"
"B--DOOM"
"C"
"D"
"E"
"F"
"G"

So, GRP_START is the start of a set and till the next GRP_START I want the records to be written to the finalfile.txt.

1

The following awk script will read through the data file and temporarily store each record in the lines variable. If the keyword in the variable word is found in the record, the do_output variable is set to 1 (true). When the keyword has been found and we either reach the next record or the end of the file, the keyword record is outputted.

#!/usr/bin/awk -f

/^"GRP_START"/ {
    if (do_output) {
        exit;
    }
    lines = $0;
    next;
}

$0 ~ word {
    do_output = 1;
}

{
    lines = sprintf("%s\n%s", lines, $0);
}

END {
    if (do_output) {
        print lines;
    }
}

Testing it:

$ awk -v word="MOOM" -f script.awk basefile.txt
"GRP_START","LINE1"........
"E"
"F"
"G"
"C--MOOM"

Then it's just a matter of looping over the wanted keywords:

#!/bin/sh

while read -r word; do
  awk -v word="$word" -f script.awk basefile.txt
done <keywords.txt >finalfile.txt

Suggestion for future improvement: Read the keywords into an array first, then parse the basefile.txt data file only once.

0

Try:

awk 'FNR==NR{re=re (re?"|":"")$0;next} /^"GRP_START"/{if (rec~re)print rec; rec=$0;next} {rec=rec"\n"$0} END{if (rec~re)print rec}' keywords.txt basefile.txt 

Example:

$ awk 'FNR==NR{re=re (re?"|":"")$0;next} /^"GRP_START"/{if (rec~re)print rec; rec=$0;next} {rec=rec"\n"$0} END{if (rec~re)print rec}' keywords.txt basefile.txt 
"GRP_START","LINE1"........
"A"
"B--BOOM"
"C"
"GRP_START","LINE1"........
"A"
"B--DOOM"
"C"
"D"
"E"
"F"
"G"
"GRP_START","LINE1"........
"E"
"F"
"G"
"C--MOOM"

How it works

  • FNR==NR{re=re (re?"|":"")$0;next}

    While reading the first file, keywords.txt, we construct a regular expression to match any of the words. For the sample data, for example, this creates the variable re with the value BOOM|DOOM|MOOM.

  • /^"GRP_START"/{if (rec~re)print rec; rec=$0;next}

    Every time we reach the start of a new record, we check to see if the last record, stored in rec matches the regular expressions stored in re. If it does, we print it. Next, rec is reset to the current line and we skip the rest of the commands and jump to the next line.

  • rec=rec"\n"$0

    For all other lines, we append them to rec.

  • END{if (rec~re)print rec}

    After we finish reading the file, we check to see if the final record matches the regex re and, if so, we print it.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.