# Averaging and moving a selection of files

I have a bunch of directories, each of them enclosing an ascii table named accordingly with the directory name; as follow:

``````a1/a1.txt
a2/a2.txt
a3/a3.txt
a4/a4.txt
b1/b1.txt
b2/b2.txt
b3/b3.txt
b4/b4.txt
``````

I would like to operate a selection over `a` and `b` 4 different files, by calculating the average value of the fourth column of each files `a` ( so `a1.txt a2.txt a3.txt a4.txt`) and `b` (so `b1.txt b2.txt b3.txt b4.txt`); and then move the table with the highest average value to another directory.

So if:

``````a1.txt average value = 1
a2.txt average value = 0.25
a3.txt average value = 2
a4.txt average value = 1.15
b1.txt average value = 3
b2.txt average value = 1.7
b3.txt average value = 0.25
b4.txt average value = 2
``````

Then `a3.txt` and `b1.txt` shall be moved to a distant directory.

I have the following code:

``````for x in a b
do
RESULT=\$(awk '{x+=\$4} END{print x/NR, FILENAME}' \${x}[1-4]/\${x}[1-4].txt | sort -n -r| head -1)
highest="\$(echo \$RESULT | cut -d ' ' -f1 )"
hifile="\$(echo \$RESULT | cut -d ' ' -f2 )"
echo "highest was \$highest in \$hifile"
cp "\$hifile" "high_dis/\${x}.txt"
done
``````

But this code just calculate the average value of the four files and then copy the last file (`a4.txt` and `b4.txt`)

Does anyone know how to obtain the desired output?

• Both answers do the job! Nov 18 '15 at 15:53

Your problem was due to the fact that only after reading all input files, awk reaches the END clause.

Replacing the first line of your `for` loop with the following shall solve your problem.

``````RESULT=\$(awk 'BEGIN{curr_f=FILENAME; nr=1} {if(curr_f!=FILENAME){if(mean > max_mean){max_mean=mean; f=curr_f} curr_f=FILENAME; nr=0; sum=0} } { sum+=\$4; nr++; mean=sum/nr }  END{if(mean>max_mean){print mean, FILENAME}else{print max_mean, f}}' \${x}[1-4]/\${x}[1-4].txt )
``````
• It seems to work. Thank you. But by now it does the job over the first column of my tables; unfortunately I need to average the fourth column (with space delimiter) of each tables. Where should I put the command to average the desired column Nov 18 '15 at 15:35
• Oh, right, it is because in my tests I was using \$1 instead of \$4, just replace this part. Ill edit my answer to reflect that.
– Kira
Nov 18 '15 at 15:38

I would use a slightly different approach and do more of the work directly in awk. First, use the `FNR` which goes back to 1 each time a new file is processed by `awk` and feed all a- or b- files to `awk` at once:

``````\$ for f in a b; do
awk '(FNR==1 && n>1){
av=x/n;if(av>max){
max=av;f=FILENAME
}
x=0; n=0
}
{x+=\$4;n=FNR}
END{
if(x/n>max){print x/n,FILENAME}
else{print max,f}'
} "\$f"[1..4]/"\$f"[1..4].txt; done
18898.4 a3.txt
18806 b4.txt
``````

That gives us the two files with the greatest average. To move them, you could simply remove the value from the previus script's output, keeping only the file name and pass that directly to the `cp` command:

``````\$ cp -v \$(for f in a b; do awk '(FNR==1 && n>1){av=x/n;if(av>max){max=av;f=FILENAME};x=0;n=0}{x+=\$4;n=FNR} END{if(x/n>max) print x/n,FILENAME; else print max,f}' "\$f"[1-4]/"\$f"[1..4].txt; done) bar/
‘a3.txt’ -> ‘bar/a3.txt’
‘b4.txt’ -> ‘bar/b4.txt’
``````

Note that this will break if your filenames contain whitespace or glob characters.

• This one also works just perfectly! Thanks Nov 18 '15 at 15:52
• I might be mistaken, but I believe that this code might not work when the greatest average is in the last file, as you only update the `max` and `f` variables when you reach a new file. In your example it seems to work though.
– Kira
Nov 18 '15 at 17:45
• Testing here, in fact it only works properly adding this in the last awk's line: `END{if(x/n>max) print x/n,FILENAME; else print max,f}`
– Kira
Nov 18 '15 at 17:49
• @Kira d'oh! Of course, otherwise it will fali if the last file processed is the one with the greatest value. Thanks, fixed.
– terdon
Nov 18 '15 at 17:58