0

I would like to return the average of all numbers after a specific string in multiple files i.e.

We are given 10 files (file1.txt,...,file10.txt). Each file contains among others

Test1: Avg. length 24.01000, time: 0.579
Test2: Avg. length 22.02000, time: 0.879

with different numbers.

If we have 10 files it could look like

file1.txt

Test1: Avg. length 24.01000, time: 0.679
Test2: Avg. length 22.01000, time: 0.479

file2.txt

Test1: Avg. length 27.01000, time: 0.279
Test2: Avg. length 24.01000, time: 0.779

.....

file10.txt

The output I desire is the mean of the lengths and times of Test1 and Test2 over all files:

Mean Test1: Avg. length (file1_Test1_length+...+file10_Test1_lenght)/10, time (file1_Test1_time+...+file_10_Test1_time)/10
Mean Test2: Avg. length (file1_Test2_length+...+file10_Test2_lenght)/10, time (file1_Test2_time+...+file_10_Test2_time)/10

To grep the full output of Test1 I execute:

egrep -rh 'Test1: Avg. length.*' /home/timo/Documents

I'm not sure how to grep only the numbers. I tried

egrep -rhP '(?<=length )\d+' /home/timo/Documents

But I get an error

grep: conflicting matchers specified

I would be really thankful if someone could help me!

2
  • If you say "among others", does that mean there are other lines besides the two you mention, but these two look exactly as reproduced by you, or that these two lines can in addition contain other comma-separated fields? If the latter, please explain what the structure of the line looks like exactly.
    – AdminBee
    Feb 16, 2021 at 10:03
  • The files contains other lines. Abcsd Test1... Test2... The lines containing Test... are always the same with exception of the numbers
    – GeoRie
    Feb 16, 2021 at 10:11

2 Answers 2

1

The following awk solution should work:

awk -F'[ ,:]+' '$1~/^Test[12]/{l[$1]+=$4; t[$1]+=$6; n[$1]++;}
                END{if (n["Test1"]) {for (tst in l) printf("Mean %s: Avg. length %f, time: %f\n",tst,l[tst]/n[tst], t[tst]/n[tst]);} else {print "No input found"}}' file*.txt

This will parse the input files for lines starting Test1 or Test2 and sum fields 4 and 6 (the "length" and "time", respectively). In addition, it will increase the data counter n. In the end, it will either print the average (if any data was found) or an error message.

If you are sure that at least one file will be present, you can simplify this to

awk -F'[ ,:]+' '$1~/^Test[12]/{l[$1]+=$4; t[$1]+=$6; n[$1]++;}
                END{for (tst in l) printf("Mean %s: Avg. length %f, time: %f\n",tst,l[tst]/n[tst], t[tst]/n[tst]);}' file*.txt

Since all your files seem to be in individual sub-folders, the approach depends on your shell. In the easiest case, you can try

awk -F'[ ,:]+' ' ... ' subdir*/file*.txt
0

Using GNU datamash:

$ grep '^Test.*Avg\. length.*time:' file*.txt | tr -d ',' | LC_ALL=C datamash -W -s -g 1 mean 4,6
Test1:  25.51   0.479
Test2:  23.01   0.629

This first extracts the lines that you show using grep. I do this by matching the text Test at the start of a line, followed by the strings Avg. length and time: somewhere else on the line. You may want to modify this expression (I don't know if it uniquely matches only the lines that we're interested in).

I then delete all commas from the data, because they mess up the interpretation of the numbers. I do this with tr.

The grep+tr bit could also be done with sed as

sed '/^Test.*Avg\. length.*time:/!d; s/,//g' file*.txt

I then use GNU datamash to compute the mean lengths and times for each test. I first tell datamash that whitespaces are to be used as delimiters using -W. The -s sorts the data, because it comes unsorted from grep+tr (or sed), to efficiently group the data globally.

Grouping is done with -g 1 which defines the TestN: labels on each line as the grouping key. Then, for each such group, the mean is calculated over the 4th and 6th whitespace-delimited column, with mean 4,6.

I set the locale to C (the POSIX locale) for datamash, because the utility may otherwise want decimal numbers to use comma rather than dot as a decimal dot.

Would you want to decorate the output a bit, do so with awk:

sed '/^Test.*Avg\. length.*time:/!d; s/,//g' file*.txt |
LC_ALL=C datamash -W -s -g 1 mean 4,6 |
awk '{ printf "%s Avg. length: %s time: %s\n", $1,$2,$3 }'

This may output something like

Test1: Avg. length: 25.51 time: 0.479
Test2: Avg. length: 23.01 time: 0.629
7
  • I forgot to say that each file is located in a subfolder. Therefore I use egrep -rh '^Test.*Avg\. length.*time:' /home/timo/Documents/ | tr -d ',' | datamash -W -s -g 1 mean 4,6 Then I get: datamash: invalid numeric value in line 1 field 4: '24.01000' Another question: Can grouping be done by names e.g. -g 'timo' if the names would not be Test1 and Test2 but 'Timo' and 'Anna'? Thanks
    – GeoRie
    Feb 16, 2021 at 9:56
  • @GeoRie I don't understand why you get a "invalid numeric value" error. That value does not look invalid to me. As for using Timo and Anna instead of Test1 and Test2: You would have to modify your grep to look for ^[[:alnum:]]*:.*Avg\. length.*time: instead. [[:alnum:]]* would match any alphanumeric string. Always show data that is actually representative of your real data.
    – Kusalananda
    Feb 16, 2021 at 10:10
  • the error also appears if I execute datamash -W -s -g 1 mean 6 datamash: invalid numeric value in line 1 field 6: '0.684'. I don't understand why '0.684' isn't a valid numeric value? Perhaps the value is from type string and must be converted into numeric?
    – GeoRie
    Feb 16, 2021 at 11:15
  • @GeoRie This makes no sense to me. Do you have control characters in the data? I had to remove the commas from each line because these were interfering, but I can se nothing else in the data in the question that would be an issue.
    – Kusalananda
    Feb 16, 2021 at 11:28
  • Without using datamash i get Test1: Avg. length 24.36000 time: 0.526 Test2: Avg. length 24.21000 time: 0.379 Test1: Avg. length 24.14000 time: 0.531 Test2: Avg. length 24.47000 time: 0.383 Test1: Avg. length 24.26000 time: 0.547 Test2: Avg. length 24.35000 time: 0.384 Test1: Avg. length 24.53000 time: 0.574 Test2: Avg. length 24.26000 time: 0.416 I saved this output in a file and try datamash -W -s -g 1 mean 4,6 </home/timo/Documents/mytest.txt Same result: datamash: invalid numeric value in line 1 field 4: '24.36000' Could you try it, too, please
    – GeoRie
    Feb 16, 2021 at 11:36

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .