2

I want to pass two variables to my AWK script from my shell script.

#!/bin/bash

HeaderSize=$(grep -n "# end header data" input.list | tr ":" "\n" | head -n 1)
RecordSize=$(grep -n "# Number of unique" input.list | tr ":" "\n" | tail -n 1 | sed 's/,//g')

echo $HeaderSize
echo $RecordSize

head -n $HeaderSize input.list > temp.list

./parse.awk -v headersize=$HeaderSize -v recordsize=$RecordSize < input.list >> temp.list

File parse.awk looks something like this:

#!/bin/gawk -f

BEGIN {
    curline=1;
    excludeline=0;
    newrecordsize=0;
}
{
    if (curline < $headersize) {
        curline++;
    } else {
        if (($2 !~ /test1/) && ($2 !~ /test2/) && ($2 !~ /test3/)) {
            print $0;
        } else {
            excludeline++;
        }
    }
}
END {
    print "#", excludeline, "lines excluded";
    newrecordsize = recordsize - excludeline;
    printf "# Number of unique items after exclusions: %'d\n", newrecordsize;
}

So I just wanted to copy the header directly over, then do some exclude some of the data.

When I run the Bash file I get an error on the gawk execution:

38
94108
gawk: ./parse.awk:6: fatal: cannot open file `94108' for reading (No such file or directory)

94108 is the value for RecordSize or the second variable I'm trying to pass.

How can I fix this error?

Edit: The answer is found below has to do with double quotes on the variables, which apparently I should be doing everywhere. Here are my current script which works fine.

#!/bin/bash

HeaderSize=$(grep -n "# end header data" input.list | tr ":" "\n" | head -n 1)
RecordSize=$(grep -n "# Number of unique" input.list | tr ":" "\n" | tail -n 1 | sed 's/,//g')

echo "$HeaderSize"
echo "$RecordSize"

head -n "$HeaderSize" input.list > temp.list

./parse.awk -v "headersize=$HeaderSize" -v "recordsize=$RecordSize" < input.list >> temp.list

I also cleaned up the awk script

#!/bin/gawk -f

BEGIN {
    excludeline=0;
    newrecordsize=0;
}

NR > headersize {
    if (($2 !~ /test1/) && ($2 !~ /test2/) && ($2 !~ /test3/)) {
        print $0;
    } else {
        excludeline++;
    }
}
END {
    print "#", excludeline, "lines excluded";
    newrecordsize = recordsize - excludeline;
    printf "# Number of unique domains after exclusions: %'d\n", newrecordsize; 
}
7
  • You are correctly passing the two values to your gawk script. Something else is at play as your scripts run without error for me.
    – fpmurphy
    Sep 25 at 5:23
  • Copy/paste your shell script into shellcheck.net, fix the issues it will tell you about, and then edit your question to show the corrected script if you still have a problem or, more likely, delete the question if you don't.
    – Ed Morton
    Sep 25 at 12:07
  • @fpmurphy it runs in bash fine but doesn't run in script. The issue has to do with putting things in quotes as per the answer below. shellcheck.net popped this up as a suggestion.
    – misosoup7
    Sep 26 at 4:50
  • 1
    @EdMorton, thanks, I will be using that going forward.
    – misosoup7
    Sep 26 at 4:51
  • You're welcome. FYI although putting the quotes around the whole assignment statement, "recordsize=$RecordSize", won't do any harm, just putting the quotes around the value in the statement is far more common syntax - recordsize="$RecordSize".
    – Ed Morton
    Sep 26 at 11:53
7

I suspect there’s whitespace at the start of $RecordSize; quote your variables:

echo "$HeaderSize"
echo "$RecordSize"

./parse.awk -v "headersize=$HeaderSize" -v "recordsize=$RecordSize" < input.list >> temp.list
1
  • Also note probable incorrect use of awk variable headersize in their code (and possible also numerous other improvements that could be made).
    – Kusalananda
    Sep 25 at 10:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.