I have a bunch of directories, each of them enclosing an ascii table named accordingly with the directory name; as follow:


I would like to operate a selection over a and b 4 different files, by calculating the average value of the fourth column of each files a ( so a1.txt a2.txt a3.txt a4.txt) and b (so b1.txt b2.txt b3.txt b4.txt); and then move the table with the highest average value to another directory.

So if:

a1.txt average value = 1
a2.txt average value = 0.25
a3.txt average value = 2
a4.txt average value = 1.15
b1.txt average value = 3
b2.txt average value = 1.7
b3.txt average value = 0.25
b4.txt average value = 2

Then a3.txt and b1.txt shall be moved to a distant directory.

I have the following code:

for x in a b
  RESULT=$(awk '{x+=$4} END{print x/NR, FILENAME}' ${x}[1-4]/${x}[1-4].txt | sort -n -r| head -1)
  highest="$(echo $RESULT | cut -d ' ' -f1 )"
  hifile="$(echo $RESULT | cut -d ' ' -f2 )"
  echo "highest was $highest in $hifile"
  cp "$hifile" "high_dis/${x}.txt"

But this code just calculate the average value of the four files and then copy the last file (a4.txt and b4.txt)

Does anyone know how to obtain the desired output?

  • Both answers do the job!
    – steve
    Nov 18 '15 at 15:53

Your problem was due to the fact that only after reading all input files, awk reaches the END clause.

Replacing the first line of your for loop with the following shall solve your problem.

RESULT=$(awk 'BEGIN{curr_f=FILENAME; nr=1} {if(curr_f!=FILENAME){if(mean > max_mean){max_mean=mean; f=curr_f} curr_f=FILENAME; nr=0; sum=0} } { sum+=$4; nr++; mean=sum/nr }  END{if(mean>max_mean){print mean, FILENAME}else{print max_mean, f}}' ${x}[1-4]/${x}[1-4].txt )
  • It seems to work. Thank you. But by now it does the job over the first column of my tables; unfortunately I need to average the fourth column (with space delimiter) of each tables. Where should I put the command to average the desired column
    – steve
    Nov 18 '15 at 15:35
  • Oh, right, it is because in my tests I was using $1 instead of $4, just replace this part. Ill edit my answer to reflect that.
    – Kira
    Nov 18 '15 at 15:38

I would use a slightly different approach and do more of the work directly in awk. First, use the FNR which goes back to 1 each time a new file is processed by awk and feed all a- or b- files to awk at once:

$ for f in a b; do 
    awk '(FNR==1 && n>1){
        x=0; n=0
            if(x/n>max){print x/n,FILENAME}
            else{print max,f}'
        } "$f"[1..4]/"$f"[1..4].txt; done
18898.4 a3.txt
18806 b4.txt

That gives us the two files with the greatest average. To move them, you could simply remove the value from the previus script's output, keeping only the file name and pass that directly to the cp command:

$ cp -v $(for f in a b; do awk '(FNR==1 && n>1){av=x/n;if(av>max){max=av;f=FILENAME};x=0;n=0}{x+=$4;n=FNR} END{if(x/n>max) print x/n,FILENAME; else print max,f}' "$f"[1-4]/"$f"[1..4].txt; done) bar/
‘a3.txt’ -> ‘bar/a3.txt’
‘b4.txt’ -> ‘bar/b4.txt’

Note that this will break if your filenames contain whitespace or glob characters.

  • This one also works just perfectly! Thanks
    – steve
    Nov 18 '15 at 15:52
  • I might be mistaken, but I believe that this code might not work when the greatest average is in the last file, as you only update the max and f variables when you reach a new file. In your example it seems to work though.
    – Kira
    Nov 18 '15 at 17:45
  • Testing here, in fact it only works properly adding this in the last awk's line: END{if(x/n>max) print x/n,FILENAME; else print max,f}
    – Kira
    Nov 18 '15 at 17:49
  • @Kira d'oh! Of course, otherwise it will fali if the last file processed is the one with the greatest value. Thanks, fixed.
    – terdon
    Nov 18 '15 at 17:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.