0

Is there an easy way to extract variables from a text file?

E.g. given the output from ab:

This is ApacheBench, Version 2.3 <$Revision: 1638069 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking bar (be patient)
Finished 1206 requests


Server Software:        Jetty(9.0.z-SNAPSHOT)
Server Hostname:        bar
Server Port:            5500

Document Path:          /foo/1
Document Length:        148 bytes

Concurrency Level:      15
Time taken for tests:   30.041 seconds
Complete requests:      1206
Failed requests:        0
Total transferred:      359686 bytes
HTML transferred:       178636 bytes
Requests per second:    40.15 [#/sec] (mean)
Time per request:       373.643 [ms] (mean)
Time per request:       24.910 [ms] (mean, across all concurrent requests)
Transfer rate:          11.69 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       47  108  36.0     98     328
Processing:    73  264 782.5    150    7951
Waiting:       73  255 721.5    148    7886
Total:        129  371 783.5    259    8039

Percentage of the requests served within a certain time (ms)
  50%    259
  66%    293
  75%    324
  80%    340
  90%    413
  95%    525
  98%    683
  99%   6421
 100%   8039 (longest request)

I'd like to extract values (matching name: value, see example below) and assign them to variables in one step. (I know that ab can export some data to csv but the rest is only available as formatted text.)

The best I've found so far is:

path=$(cat text|grep 'Document Path:'|awk -F: '{ split($2, z, " "); print z[1]}')
total=$(cat text|grep 'Total transferred:'|awk -F: '{ split($2, z, " "); print z[1]}')
#[...]

But this seems to be a bit repetitive and awkward - is there an easier way or a tool that's better suited for this job?

  • extract what values ? – 123 Jan 12 '16 at 15:05
  • Potentially all of them. – laktak Jan 12 '16 at 15:08
  • Please edit your question and make it specific. What values do you want? What is a "value"? Is ApacheBench a value? Is 50%? Or should it be 50? Show us a specific example and the output you would like to see. As it stands, this is just too broad. And yes, of course there are easy ways to do this but it all depends on what you want to extract. – terdon Jan 12 '16 at 15:29
  • @terdon I did include samples for the values - see path and total. – laktak Jan 14 '16 at 9:57
  • Yes, but those are just two. I think maybe you mean that you want all strings that end is colons (foo:) to be variable names. Is that correct? – terdon Jan 14 '16 at 10:02
2

I would look for all lines that have 1-4 words and then a :, replace the spaces between the words with underscores and print them as variable=value pairs. You can then pass the entire thing through eval to set them. For example:

$ awk -F': *' '/^(\S+\s*){1,4}:/{gsub(/ /,"_",$1);print $1"=\""$2"\""}' file
Server_Software="Jetty(9.0.z-SNAPSHOT)"
Server_Hostname="bar"
Server_Port="5500"
Document_Path="/foo/1"
Document_Length="148 bytes"
Concurrency_Level="15"
Time_taken_for_tests="30.041 seconds"
Complete_requests="1206"
Failed_requests="0"
Total_transferred="359686 bytes"
HTML_transferred="178636 bytes"
Requests_per_second="40.15 [#/sec] (mean)"
Time_per_request="373.643 [ms] (mean)"
Time_per_request="24.910 [ms] (mean, across all concurrent requests)"
Transfer_rate="11.69 [Kbytes/sec] received"
Connect="47  108  36.0     98     328"
Processing="73  264 782.5    150    7951"
Waiting="73  255 721.5    148    7886"
Total="129  371 783.5    259    8039"

The -F': * sets the field separator to : followed by 0 or more spaces. The script then checks whether this line matches 1 to 4 occurrences of a string of non-space characters ("words") followed by 0 or more spaces and then a :. I use 4 because of this line:

Time taken for tests:   30.041 seconds

Then, for lines that match, replace all spaces in the first field with underscores (gsub(/ /,"_",$1)), then print the first field, an = and the second field quoted. Because bare strings need to be quoted for awk to print them, in order to print a quoted $2, the quotes need to be escaped: " \"".

If that produces the output you want, you can now use eval to read the variables:

$ eval $(awk -F': *' '/^(\S+\s*){1,4}:/{gsub(/ /,"_",$1);print $1"=\""$2"\""}' file)
$ echo $Transfer_rate 
11.69 [Kbytes/sec] received

Or, just source it:

. <(awk -F': *' '/^(\S+\s*){1,4}:/{gsub(/ /,"_",$1);print $1"=\""$2"\""}' file)

Important: This can be dangerous. eval or sourcing the file will simply execute any code you give it. It will not check whether it is dangerous. If the awk script returns something dangerous like rm ~/* for some reason, then eval will happily run it. So, always check what output you get before running the above. This is also the case for the accepted answer. It is always dangerous to blindly execute code returned by another program.


The above will work for GNU awk but not for simpler awk implementatins. If it doesn't work on your system, try this instead:

. <(awk -F': *' '/.*\s*:   *:/{gsub(/ /,"_",$1);print $1"=\""$2"\""}' file)
  • That's a neat solution. Thanks for the explanation! – laktak Jan 14 '16 at 13:25
  • Do I need a special version of awk for this? I get no output with your sample. – laktak Jan 14 '16 at 13:54
  • It works with awk -F': *' '(NF == 2) && ($1 ~ /^[A-Za-z0-9 ]+$/) {gsub(/ /,"_",$1);print $1"=\""$2"\""}' though. – laktak Jan 14 '16 at 13:54
  • @laktak there shouldn't be any output when you . $(awk ...) or eval $(awk ...). Does the awk command by itself return anything? I mean just awk -F': *' '/^(\S+\s*){1,4}:/{gsub(/ /,"_",$1);print $1"=\""$2"\""}' file. If not, please ping me (@terdon) in /dev/chat and we can figure it out. – terdon Jan 14 '16 at 14:03
  • No, awk -F': *' '/^(\S+\s*){1,4}:/{gsub(/ /,"_",$1);print $1"=\""$2"\""}' file gives me no ouput (Ubuntu 15.10) – laktak Jan 14 '16 at 21:35
4

I generally use the following pattern:

. <(
    awk 'BEGIN{print "shellvarname=\"value\""}'
)

This uses awk to generate some statements that would work in the shells variable assignment syntax. This result is sourced (.).

In your specific request, this would be an option:

. <(
    awk -F': *' '
      /Document Path/{printf "%s=\"%s\"\n", "path", $2}
      /Total transferred/{printf "%s=\"%s\"\n", "total", $2}
    ' file
)

or a bit shorter

. <(
    awk '
      /Document Path/{printf "%s=\"%s\"\n", "path", $3}
      /Total transferred/{printf "%s=\"%s\"\n", "total", $3}
    ' file
)
  • . <( command ) is an interesting hack and I can think of occasions where it would be very useful. I think parsing ab's formatted-for-display output this way is probably a very fragile solution, though. – Wildcard Jan 12 '16 at 15:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.