Extract Values from Formatted Text

Question

Is there an easy way to extract variables from a text file?

E.g. given the output from ab:

This is ApacheBench, Version 2.3 <$Revision: 1638069 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking bar (be patient)
Finished 1206 requests


Server Software:        Jetty(9.0.z-SNAPSHOT)
Server Hostname:        bar
Server Port:            5500

Document Path:          /foo/1
Document Length:        148 bytes

Concurrency Level:      15
Time taken for tests:   30.041 seconds
Complete requests:      1206
Failed requests:        0
Total transferred:      359686 bytes
HTML transferred:       178636 bytes
Requests per second:    40.15 [#/sec] (mean)
Time per request:       373.643 [ms] (mean)
Time per request:       24.910 [ms] (mean, across all concurrent requests)
Transfer rate:          11.69 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       47  108  36.0     98     328
Processing:    73  264 782.5    150    7951
Waiting:       73  255 721.5    148    7886
Total:        129  371 783.5    259    8039

Percentage of the requests served within a certain time (ms)
  50%    259
  66%    293
  75%    324
  80%    340
  90%    413
  95%    525
  98%    683
  99%   6421
 100%   8039 (longest request)

I'd like to extract values (matching name: value, see example below) and assign them to variables in one step. (I know that ab can export some data to csv but the rest is only available as formatted text.)

The best I've found so far is:

path=$(cat text|grep 'Document Path:'|awk -F: '{ split($2, z, " "); print z[1]}')
total=$(cat text|grep 'Total transferred:'|awk -F: '{ split($2, z, " "); print z[1]}')
#[...]

But this seems to be a bit repetitive and awkward - is there an easier way or a tool that's better suited for this job?

Answer 1

I generally use the following pattern:

. <(
    awk 'BEGIN{print "shellvarname=\"value\""}'
)

This uses awk to generate some statements that would work in the shells variable assignment syntax. This result is sourced (.).

In your specific request, this would be an option:

. <(
    awk -F': *' '
      /Document Path/{printf "%s=\"%s\"\n", "path", $2}
      /Total transferred/{printf "%s=\"%s\"\n", "total", $2}
    ' file
)

or a bit shorter

. <(
    awk '
      /Document Path/{printf "%s=\"%s\"\n", "path", $3}
      /Total transferred/{printf "%s=\"%s\"\n", "total", $3}
    ' file
)

Answer 2

I would look for all lines that have 1-4 words and then a :, replace the spaces between the words with underscores and print them as variable=value pairs. You can then pass the entire thing through eval to set them. For example:

$ awk -F': *' '/^(\S+\s*){1,4}:/{gsub(/ /,"_",$1);print $1"=\""$2"\""}' file
Server_Software="Jetty(9.0.z-SNAPSHOT)"
Server_Hostname="bar"
Server_Port="5500"
Document_Path="/foo/1"
Document_Length="148 bytes"
Concurrency_Level="15"
Time_taken_for_tests="30.041 seconds"
Complete_requests="1206"
Failed_requests="0"
Total_transferred="359686 bytes"
HTML_transferred="178636 bytes"
Requests_per_second="40.15 [#/sec] (mean)"
Time_per_request="373.643 [ms] (mean)"
Time_per_request="24.910 [ms] (mean, across all concurrent requests)"
Transfer_rate="11.69 [Kbytes/sec] received"
Connect="47  108  36.0     98     328"
Processing="73  264 782.5    150    7951"
Waiting="73  255 721.5    148    7886"
Total="129  371 783.5    259    8039"

The -F': * sets the field separator to : followed by 0 or more spaces. The script then checks whether this line matches 1 to 4 occurrences of a string of non-space characters ("words") followed by 0 or more spaces and then a :. I use 4 because of this line:

Time taken for tests:   30.041 seconds

Then, for lines that match, replace all spaces in the first field with underscores (gsub(/ /,"_",$1)), then print the first field, an = and the second field quoted. Because bare strings need to be quoted for awk to print them, in order to print a quoted $2, the quotes need to be escaped: " \"".

If that produces the output you want, you can now use eval to read the variables:

$ eval $(awk -F': *' '/^(\S+\s*){1,4}:/{gsub(/ /,"_",$1);print $1"=\""$2"\""}' file)
$ echo $Transfer_rate 
11.69 [Kbytes/sec] received

Or, just source it:

. <(awk -F': *' '/^(\S+\s*){1,4}:/{gsub(/ /,"_",$1);print $1"=\""$2"\""}' file)

---

Important: This can be dangerous. eval or sourcing the file will simply execute any code you give it. It will not check whether it is dangerous. If the awk script returns something dangerous like rm ~/* for some reason, then eval will happily run it. So, always check what output you get before running the above. This is also the case for the accepted answer. It is always dangerous to blindly execute code returned by another program.

---

The above will work for GNU awk but not for simpler awk implementatins. If it doesn't work on your system, try this instead:

. <(awk -F': *' '/.*\s*:   *:/{gsub(/ /,"_",$1);print $1"=\""$2"\""}' file)