Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a data file that I want to normalize using awk, based on the last datapoint. Therefor, I would like to access the last data point first, to normalize the data, then process normally.

The following method, using tac twice, does the job, but, is maybe more complicated than necessary.

$ cat file
0 5
1 2
2 3
3 4
$ tac file | awk 'NR==1{norm=$2} {print $1, $2/norm}' | tac
0 1.25
1 0.5
2 0.75
3 1

My question is the following: Is it possible to obtain the above result by using awk only?

I think the answer is "No, awk scans the file line by line", but I am open for suggestions for alternatives.

share|improve this question
add comment

3 Answers

up vote 5 down vote accepted

You can do it as a two-pass solution in awk:

awk 'FNR == NR { n = $2; next } { print $1, $2/n }' infile infile

If your version of awk supports the ENDFILE block (e.g. GNU awk 4+), you can do it like this:

awk 'ENDFILE { n = $2 } FNR != NR { print $1, $2/n }' infile infile

Note that it is more efficient to seek to the end of the file first see camh's answer.

Explanation

The first example works by remembering the previous $2, i.e. it is only evaluated when the local line counter (FNR) is equal to the global line counter (NR). The next command skips to the next line, in this case it ensures that the last block is only evaluated when the second argument is parsed.

The second example has similar logic, but takes advantage of the the ENDFILE block which is evaluated when the end of an input-file is reached.

share|improve this answer
    
First example does work fine, second does not $ awk --version GNU Awk 3.1.8. Can you maybe add a very small explanation on how two input file are handled and what next does? –  Bernhard Dec 19 '12 at 8:16
1  
@Bernhard: see edit –  Thor Dec 19 '12 at 8:33
add comment

If your data source is a file that can be read multiple times (i.e. it is not a stream), you should first use tail(1) to get the data you want from the last line and pass that to awk for its sequential processing of the file. tail will seek to the end of the file to read the last line without needing to read all the data before it.

awk -v norm=$(tail -n 1 file | cut -d' ' -f2) '{print $1, $2/norm}' file

This will be a big win on large files where the whole file will not fit in the buffer cache (meaning it would need to be read from disk twice, once for each pass), and will help to a smaller extent by not needing to scan the input to get to the last line. Smaller files may not show much difference to a two-pass approach.

share|improve this answer
add comment

You could load them into an array and read it backwards:

awk '{x[i++]=$0} END{for (j=i-1; j>=0;) print x[j--] }'

You could do it more efficiently, but this kind of illustrates why awk is not the right tool for this. Continue using tac where available, GNU tac is generally the fastest out of a variety of tools for this job.

share|improve this answer
    
I agree, using a for-loops in awk is not the solution. –  Bernhard Dec 19 '12 at 8:17
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.