I want to loop through all the files in the directory.

the files are set out like this:

other data
other data

I have the code:

for file in .dat; 
awk 'x+=sub(/<Overall>/,""){y+=$0} END{print FILENAME, y/x}' $file

this prints out the average of the values in the file, however what I want is to take the directory my script is in as an argument and do the awk command on all the .dat files in the directory.

I have tried using the code:

for file in $1

but get the error:

awk: cmd. line:1: fatal cannot open file `folder' for reading (No such file or directory)

In addition to this I also want to be able to sort the output of average values from high to low.

  • 1
    for file in *.dat ? – Jeff Schaller Feb 15 '16 at 21:37
  • for file in $1/* ? – Archemar Feb 16 '16 at 9:59
  • for file in $1/*.dat – Alex Aug 23 '17 at 13:44

Two variations:

  1. Loop over the files and invoke awk once for each file, or
  2. Give the awk script all files and let it calculate the average for each and report as it goes along.

Sorting of the result of any of the below solutions may be done by piping their output through

sort -k2,2rn

This does a reverse numeric sort on the second field (the averages).

First solution:


for name in "$1"/*.dat; do
    test -f "$name" || continue   # skip non-files
    awk -F '>' '/<Overall>/ { s+=$NF; n++ } END { print FILENAME, s/n }' "$name"

This script expects a directory name on the command line as the first and only command line argument. The awk script will find all lines containing the string Overall, and will sum up (in s) the value after the > on that line. At the end, the average is outputted together with the filename. The variable n holds the number of times we've added something to s.

Second solution (requires GNU Awk):


find "$1" -maxdepth 1 -type f -name '*.dat' \
    -exec awk -F '>' '/<Overall>/ { s+=$NF; n++ } ENDFILE { print FILENAME, s/n; s=n=0 }' {} +

This script, like the first, expects a directory name as its only command line argument. It uses find to execute an awk script with as many .dat files as possible at time.

The awk script makes use of GNU Awk's ENDFILE trigger to output the calculated values and reset the s and n variables after processing each file, before starting to read the next file.

This may also have been written as


awk -F '>' '/<Overall>/ { s+=$NF; n++ } ENDFILE { print FILENAME, s/n; s=n=0 }' "$1"/*.dat

but this relies on "$1"/*.dat not expanding to a too long list of filenames (this also requires each .dat name to be a regular file, which is something that the above find command guarantees with -type f).

  • It seems that by clever use of a FNR == 1 test, it should be possible to make this portable to any POSIX Awk. In other words, you can do without the ENDFILE feature. – Wildcard Sep 19 '17 at 21:38
  • 1
    @Wildcard That is certainly possible. You'll get some duplication of code (for output in FNR==1 {} and in END {}) so it becomes slightly less straightforward. I'll leave it like this for now as the two ways are to a large extent identical with ENDFILE in the second variation more or less matching END in the first variation. – Kusalananda Sep 19 '17 at 21:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.