1

I've been tasked to create a list whereby one of the codes in the original data is to be replaced by a new code read in from a reference list. In this case there is just one change, but there could be more which would be added to the reference list as and when needed.

The reference list (mycodes) has the following value:

100,100007

The data is a stream of three digit codes, but the code for 100 should be written out along with the rest of the stream as a five digit code. I have used an AWK program as follows;-

BEGIN{
FS=","
reffile="mycodes"
while(getline<reffile>0) {ref[$1]=$2}
}
{
val=$1
newval=ref[val]

if (newval in ref) { outval=val}
else               {outval=newval}

print outval
}

With the input data file containing the following values:

100
101
120
130
100

the program when run does produce the correct output of

100007
101
120
130
10007

However it only works if there is a space in the reference file after the first entry. If the space is missing then the program does not produce anything other than 100007 as output.

I don't understand exactly what is happening in the logic of this AWK program and I was wondering if somebody could helpfully explain it - particularly the line about if (newval in ref).

7
  • 1
    Please edit your post to include an example extract of both input files and the desired output; the task is much easier to understand that way.
    – AdminBee
    Commented May 21, 2021 at 14:59
  • 1
    With those inputs, why would the program ever produce 10007?
    – Kusalananda
    Commented May 21, 2021 at 15:01
  • 1
    Does "if there is a space in the reference file after the first entry" actually mean a blank line in the file? Or a literal space character? The former would explain the result you are seeing - except for the 100007 v. 10007 discrepancy that Kusalananda pointed out.
    – fra-san
    Commented May 21, 2021 at 15:03
  • Sorry, just to clarify, if the reference file has a blank line after the first entry then the output is correct, if the reference file only contains the value 100,100007 then the output only produces 100007 to match the input data and blank lines when there is no match up
    – Jonathan B
    Commented May 21, 2021 at 15:07
  • Is it possible your files were created on a Windows system? The presence of a CR (\r) causes your output to a terminal to overwrite, which can obscure your visible results. Commented May 21, 2021 at 15:25

4 Answers 4

1

If you're going to use getline to populate $0 then here's how to do it (see http://awk.freeshell.org/AllAboutGetline):

while ( (getline < reffile) > 0 ) {
    ref[$1] = $2
}

and the rest of your script should just be one line:

{ print ( $1 in ref ? ref[$1] : $1 ) }

so the entire script is:

BEGIN {
    FS = ","
    reffile = "mycodes"
    while ( (getline < reffile) > 0 ) {
        ref[$1] = $2
    }
}
{ print ( $1 in ref ? ref[$1] : $1 ) }

I assume you have a good reason for creating a variable to hold the file name "mycodes" and don't want to pass it in as an argument.

Alternatively you could just do:

BEGIN { FS = "," }
NR==FNR { ref[$1] = $2; next }
{ print ( $1 in ref ? ref[$1] : $1 ) }

and call it as awk 'script' mycodes file. That's slightly less efficient than populating refs[] in the BEGIN with a getline loop but that's unlikely to be an issue for most applications and obviously uses more concise and much-harder-to-get-wrong code.

Doing print ( $1 in ref ? ref[$1] : $1 ) is a bit more efficient than doing if ($1 in ref) $1=ref; print $1 or similar because it doesn't force awk to rebuild the current record but, again, unlikely to be significant.

Having said that, though it has issues, your existing script probably won't fail in the ways you're describing and your real problem is DOS line endings (see https://stackoverflow.com/questions/45772525/why-does-my-tool-output-overwrite-itself-and-how-do-i-fix-it?).

0
val=$1
newval=ref[val]
if (newval in ref) { outval=val }
else               { outval=newval }

So, you read val from the main input file, and those are values like 100 or 123. With ref containing pairs like ref[100]=100007, you probably want to check if val exists as a key in ref (i.e., if the element ref[val] exists). And not if ref[val] exists as a key (or ref[ref[val]] as the element).

So make that if (val in ref).

Then, if it does exist, you probably want to use the value found from there (the one you have in newval), and if not, then the old value (val). So make that

val=$1
newval=ref[val]
if (val in ref) { outval=newval }
else            { outval=val }

Except that as @αғsнιη mentions, now you have the problem that the assignment newval=ref[val] creates ref[val] with the empty string as a value, if it didn't already exist, so we need to do something about that.

Either remove newval altogether and use ref[val] only after the test:

val=$1
# newval=ref[val]     # remove this line
if (val in ref) { outval=ref[val] }
else            { outval=val }

Or, leave it there, and test it against the empty string:

val=$1
newval=ref[val]
if (newval != "") { outval=newval }
else              { outval=val }

The difference between those is meaningful if mycodes can contain empty values in the second field.

0
0

Given

reffile="mycodes"
while(getline<reffile>0) {ref[$1]=$2}

an empty line in mycodes creates an element of ref whose value is the empty string and whose index is, again, the empty string.

Then, when processing the input stream, at

newval=ref[val]

newval is assigned the empty string every time $1 (100, 101, 120, 130) is not 100, the only index in ref whose element is not empty (the indexes are the initial 100 and "" (empty), plus those of the empty elements subsequntly created by each newval=ref[val], i.e. 101, 120, 130). In those cases, since the empty string is one of the indexes of ref, in

if (newval in ref) { outval=val}
else               {outval=newval}

if (newval in ref) succeeds and the printed value is the current value from the input stream (val, from val=$1).

On the other hand, if ref has no element with the empty string as index (as it happens when there is no blank line in mycodes), then if (newval in ref) fails every time newval is the empty string. And then newval (an empty line), is printed.

In both cases, when $1 is 100, newval is assigned the value 10007; if (newval in ref) fails and newval is hence printed.

Note that the same puzzling behavior could be triggered by an empty line in the input stream even if there was no empty line in the mycodes file.

Assuming that, in your question, one or more of 10007, 100007 and "five digit code" is a typo, and you actually always mean 10007 (five digits), I would rewrite your AWK program as:

awk -v FS=, '
  NR == FNR {
    ref[$1] = $2
    next
  }
  ($1 in ref) {
    $1 = ref[$1]
  }
  1
' ./mycodes -

or

awk '
  BEGIN {
    FS=","
    while ( (getline < "mycodes") > 0 )
      ref[$1] = $2
  }
  ($1 in ref) {
    $1 = ref[$1]
  }
  1
' -

(Thanks to αғsнιη for pointing out that newval=ref[val] actually adds val as an index to ref and that ($1 in ref) is hence a safer way to search the array for a matching index).

0
0

The main problem is on this line newval=ref[val], where the variable val as the key (equal to $1 in val=$1) which then that actually means newval=ref[$1], with this line, the newval variable's content sets to the value of the ref[val] from the ref[] array if that key was exist, otherwise it sets to empty value; doing this way, i.e newval=ref[val] if the key was not found in the array, that key will be added into the array with empty value which you may encounter some future issues when you didn't want your array size/indexes to be changed or it might exceed the available memory or significantly slow the script down if that file is huge.

... then in the if-else statement and you are testing on the newval, it always runs the else part.

if (newval in ref) {
        outval = val;    ## this section never runs
} else {
        outval = newval  ## this sections runs for every tests
}

so whenever in the line newval=ref[val], the key was exist, outval will take the value of that key, else that would be set to an empty value. and by printing in print outval, that will output those key's value if they were found in the ref[] array, or empty line otherwise.

simple fix (without extra improvements), would be changing that line into this:

newval = (val in ref)?ref[val]:val

Your command all could be written as following:

$ awk 'BEGIN{ FS="," }
    NR==FNR   { ref[$1]=$2; next }
   ($1 in ref){ $1=ref[$1] }1' reference infile
100007
101
120
130
100007

$ cat reference
100,100007
$ cat infile
100
101
120
130
100
0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .