3

I have a script which analyzes the output of a computation software. sometimes the output comes with some extra data that are irrelevant to my script. This data can be huge and makes running my simple script become seriously slow.

My script is in awk/bash.

I was wondering if it's possible to tell awk to completely ignore the lines after a specific pattern.

for example:

GOOD STUFF
----------------
IRRELEVENT DATA
----------------
IGNORE ALL THESE
----------------
END OF IT
----------------
GOOD STUFF

I was also wondering if I tell awk to look for lines starting with a specific pattern, would it ignore whatever comes after and speed up the script?

  • 4
    Just ignore those lines... e.g. /pattern1/,/pattern2/{next}but please post expected result or try to describe more accurately what you're trying to do there. – don_crissti May 31 '16 at 17:06
4

To ignore some lines on a line-by-line basis, add /unwanted pattern/ {next} or ! /wanted pattern/ {next} at the beginning of the script.

Alternatively, filter with grep: grep -v 'unwanted pattern' | awk … or grep 'wanted pattern' | awk …. This may be faster if grep eliminates a lot of lines, because grep is typically faster than awk for the same task (grep is more specialized so it can be optimized for its task; awk is a full programming language, it can do a lot more but it's less efficient).

If you want to ignore a block of consecutive lines, awk has a convenient facility for that: add /^IRRELEVENT DATA/../^END/ {next} at the top of the script to ignore all lines starting with IRRELEVENT DATA (sic) and the following lines until the first line that starts with END. You can't do that with grep; you can do it with sed (sed '/^IRRELEVENT DATA/,/^END/' | awk …) but it's less likely to be a performance gain than grep.

2

Without using next, using negation instead.

input:

$ cat f.txt 
GOOD STUFF
----------------
IRRELEVENT DATA
----------------
IGNORE ALL THESE
----------------
END OF IT
----------------
GOOD STUFF

I want to ignore lines starting with string IRRELEVENT or IGNORE or END:

$ awk   '!/IRRELEVENT|IGNORE|END/{print }' <(cat f.txt)
GOOD STUFF
----------------
----------------
----------------
----------------
GOOD STUFF

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.