Process last line first using awk

Question

I have a data file that I want to normalize using awk, based on the last datapoint. Therefor, I would like to access the last data point first, to normalize the data, then process normally.

The following method, using tac twice, does the job, but, is maybe more complicated than necessary.

$ cat file
0 5
1 2
2 3
3 4
$ tac file | awk 'NR==1{norm=$2} {print $1, $2/norm}' | tac
0 1.25
1 0.5
2 0.75
3 1

My question is the following: Is it possible to obtain the above result by using awk only?

I think the answer is "No, awk scans the file line by line", but I am open for suggestions for alternatives.

Answer 1

If your data source is a file that can be read multiple times (i.e. it is not a stream), you should first use tail(1) to get the data you want from the last line and pass that to awk for its sequential processing of the file. tail will seek to the end of the file to read the last line without needing to read all the data before it.

awk -v norm=$(tail -n 1 file | cut -d' ' -f2) '{print $1, $2/norm}' file

This will be a big win on large files where the whole file will not fit in the buffer cache (meaning it would need to be read from disk twice, once for each pass), and will help to a smaller extent by not needing to scan the input to get to the last line. Smaller files may not show much difference to a two-pass approach.

Answer 2

You can do it as a two-pass solution in awk:

awk 'FNR == NR { n = $2; next } { print $1, $2/n }' infile infile

If your version of awk supports the ENDFILE block (e.g. GNU awk 4+), you can do it like this:

awk 'ENDFILE { n = $2 } FNR != NR { print $1, $2/n }' infile infile

Note that it is more efficient to seek to the end of the file first see camh's answer.

Explanation

The first example works by remembering the previous $2, i.e. it is only evaluated when the local line counter (FNR) is equal to the global line counter (NR). The next command skips to the next line, in this case it ensures that the last block is only evaluated when the second argument is parsed.

The second example has similar logic, but takes advantage of the the ENDFILE block which is evaluated when the end of an input-file is reached.

Answer 3

You could load them into an array and read it backwards:

awk '{x[i++]=$0} END{for (j=i-1; j>=0;) print x[j--] }'

You could do it more efficiently, but this kind of illustrates why awk is not the right tool for this. Continue using tac where available, GNU tac is generally the fastest out of a variety of tools for this job.