Regex that grep numbers after specific string in multiple files and return average

Question

I would like to return the average of all numbers after a specific string in multiple files i.e.

We are given 10 files (file1.txt,...,file10.txt). Each file contains among others

Test1: Avg. length 24.01000, time: 0.579
Test2: Avg. length 22.02000, time: 0.879

with different numbers.

If we have 10 files it could look like

file1.txt

Test1: Avg. length 24.01000, time: 0.679
Test2: Avg. length 22.01000, time: 0.479

file2.txt

Test1: Avg. length 27.01000, time: 0.279
Test2: Avg. length 24.01000, time: 0.779

.....

file10.txt

The output I desire is the mean of the lengths and times of Test1 and Test2 over all files:

Mean Test1: Avg. length (file1_Test1_length+...+file10_Test1_lenght)/10, time (file1_Test1_time+...+file_10_Test1_time)/10
Mean Test2: Avg. length (file1_Test2_length+...+file10_Test2_lenght)/10, time (file1_Test2_time+...+file_10_Test2_time)/10

To grep the full output of Test1 I execute:

egrep -rh 'Test1: Avg. length.*' /home/timo/Documents

I'm not sure how to grep only the numbers. I tried

egrep -rhP '(?<=length )\d+' /home/timo/Documents

But I get an error

grep: conflicting matchers specified

I would be really thankful if someone could help me!

Answer 1

The following awk solution should work:

awk -F'[ ,:]+' '$1~/^Test[12]/{l[$1]+=$4; t[$1]+=$6; n[$1]++;}
                END{if (n["Test1"]) {for (tst in l) printf("Mean %s: Avg. length %f, time: %f\n",tst,l[tst]/n[tst], t[tst]/n[tst]);} else {print "No input found"}}' file*.txt

This will parse the input files for lines starting Test1 or Test2 and sum fields 4 and 6 (the "length" and "time", respectively). In addition, it will increase the data counter n. In the end, it will either print the average (if any data was found) or an error message.

If you are sure that at least one file will be present, you can simplify this to

awk -F'[ ,:]+' '$1~/^Test[12]/{l[$1]+=$4; t[$1]+=$6; n[$1]++;}
                END{for (tst in l) printf("Mean %s: Avg. length %f, time: %f\n",tst,l[tst]/n[tst], t[tst]/n[tst]);}' file*.txt

Since all your files seem to be in individual sub-folders, the approach depends on your shell. In the easiest case, you can try

awk -F'[ ,:]+' ' ... ' subdir*/file*.txt

Answer 2

Using GNU datamash:

$ grep '^Test.*Avg\. length.*time:' file*.txt | tr -d ',' | LC_ALL=C datamash -W -s -g 1 mean 4,6
Test1:  25.51   0.479
Test2:  23.01   0.629

This first extracts the lines that you show using grep. I do this by matching the text Test at the start of a line, followed by the strings Avg. length and time: somewhere else on the line. You may want to modify this expression (I don't know if it uniquely matches only the lines that we're interested in).

I then delete all commas from the data, because they mess up the interpretation of the numbers. I do this with tr.

The grep+tr bit could also be done with sed as

sed '/^Test.*Avg\. length.*time:/!d; s/,//g' file*.txt

I then use GNU datamash to compute the mean lengths and times for each test. I first tell datamash that whitespaces are to be used as delimiters using -W. The -s sorts the data, because it comes unsorted from grep+tr (or sed), to efficiently group the data globally.

Grouping is done with -g 1 which defines the TestN: labels on each line as the grouping key. Then, for each such group, the mean is calculated over the 4th and 6th whitespace-delimited column, with mean 4,6.

I set the locale to C (the POSIX locale) for datamash, because the utility may otherwise want decimal numbers to use comma rather than dot as a decimal dot.

Would you want to decorate the output a bit, do so with awk:

sed '/^Test.*Avg\. length.*time:/!d; s/,//g' file*.txt |
LC_ALL=C datamash -W -s -g 1 mean 4,6 |
awk '{ printf "%s Avg. length: %s time: %s\n", $1,$2,$3 }'

This may output something like

Test1: Avg. length: 25.51 time: 0.479
Test2: Avg. length: 23.01 time: 0.629