Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have this syntax in a file (http response times from analog):

<thead><tr><th class="x">seconds</th><th class="R">reqs</th><th class="r">%reqs</th><th class="B">Gbytes</th><th class="b">%bytes</th></tr></thead>
<tbody><tr><td class="x">0</td><td class="R">10927</td><td class="r"> 0.47%</td><td class="B">0.01</td><td class="b"> 0.18%</td></tr>
<tr><td class="x">&lt;= 0.01</td><td class="R">1026471</td><td class="r">44.59%</td><td class="B">0.11</td><td class="b"> 1.81%</td></tr>
<tr><td class="x">0.01-0.02</td><td class="R">535390</td><td class="r">23.26%</td><td class="B">0.06</td><td class="b"> 0.95%</td></tr>
<tr><td class="x">0.02-0.05</td><td class="R">93298</td><td class="r"> 4.05%</td><td class="B">0.27</td><td class="b"> 4.29%</td></tr>

etc.

What I want to be left with is the value for seconds - so 2 characters after "x" and before the first <

And also the request length, so 2 characters after "R" and before the next subsequent <

Probably not the best exercise to get to grips with regex but that's what I'm stuck with. Any help would be incredibly useful.

Expected result:

seconds reqs 
0 10927 
&lt= 0.01 1026471 
0.01-0.02 535390 
0.02-0.05 93298
share|improve this question
1  
The implementation of your theory: sed 's/.*"x".\([^<]*\).*"R".\([^<]*\).*/\1\t\2/' file Note that I posted this for your education only. Better do not use such solution. –  manatwork Mar 25 '13 at 7:05
4  
I feel that this may be relevant here: stackoverflow.com/questions/1732348/… ;). No but seriously, if you need to parse HTML you might want to look at html parsers. –  Kotte Mar 25 '13 at 7:08
add comment

3 Answers 3

Unless you are very certain about the format of the HTML, you have control over it, it is not critical with errors, etc. you could use regex, - but as mentioned - not recommended.

I use it frequently myself, but then usually for one time extraction of some simple data.


You could use e.g. Perl with HTML::TokeParser::Simple.

Very simplified:

#!/usr/bin/perl

use strict;
use warnings;

use HTML::TokeParser::Simple;
use HTML::Entities;
use utf8;

die "$0 [file | url]\n" unless defined $ARGV[0];

my $tp;
if ($ARGV[0] =~ /^http:\/\//) {
    $tp = HTML::TokeParser::Simple->new(url => $ARGV[0]);
} else {
    $tp = HTML::TokeParser::Simple->new(file => $ARGV[0]);
}

if (!$tp) {
    die "No HTML file found.\n";
}

# Array to store data.
my @val;
# Index
my $i = 0;

# A bit mixed code with some redundancy. 
# Could be done much simpler, - or much more safe. 
# E.g. Check for thead, tbody etc and call a sub to parse those.
# You could off course also print directly (not save to array),
# but you might want to use the data for something?
while (my $token = $tp->get_token) {
    if ($token->is_start_tag('th') && $token->get_attr('class') eq 'x') {
        $val[$i++] = $tp->get_token->as_is;
    } elsif ($token->is_start_tag('th') && $token->get_attr('class') eq 'R') {
        $val[$i++] = $tp->get_token->as_is;
    } elsif ($token->is_start_tag('td') && (
            ($token->get_attr('class') eq 'x') ||
            ($token->get_attr('class') eq 'R'))) {
        $val[$i++] = decode_entities($tp->get_token->as_is);
    }
}

my @width_col = (10, 8);

if ($i > 2 && !($i % 2)) {
    $i = 0;
    printf("%*s %*s\n",
        $width_col[0], "$val[$i++]",
        $width_col[1], "$val[$i++]"
    );
    while ($i < $#val) {
        printf("%*s %*d\n",
            $width_col[0], "$val[$i++]",
            $width_col[1], "$val[$i++]"
        );
    }
} else {
    die "ERR. Unable to extract data.\n"
}

Sample result:

$ ./extract htmlsample 
   seconds     reqs
         0    10927
   <= 0.01  1026471
 0.01-0.02   535390
 0.02-0.05    93298
share|improve this answer
add comment

As has been mentioned regex is not good for parsing html. Similar to another parse answer you can make a Ruby one-liner such as the following to do it for you. Note that it requires Nokogiri which you can install as a gem (sudo gem install nokogiri).

ruby -rnokogiri -e 'h = Nokogiri::HTML(readlines.join); h.css("tr .x").zip(h.css("tr .R")).each { |d| puts "#{d[0].content} #{d[1].content}" }' sample.html

It reads from sample.html and makes a two dimensional array that contains everything with the attribute class="x" inside a tr element paired with everything with the attribute class="R" inside a tr element. Then it prints one such pair per line. For your example the output is the following:

seconds reqs
0 10927
<= 0.01 1026471
0.01-0.02 535390
0.02-0.05 93298
share|improve this answer
add comment

This uses sed and then you can use cut to get the fields you want. This is a one-liner, but I wrote as a script file with comments for clarity.

#!/bin/sed -f
s!</*thead!<tbody!g;      # to not get caught by 'th' below
s!<t[dh][^>]*>!%%%!g;     # replace start tag 'td' or 'th' with a delimitor
s!</t[dh]>!@@@!g;         # replace end tag 'td' or 'th' with a delimitor
s/<[^>]*>//g;             # delete any other tags
s/%%%\([^@]*\)@@@/\1 /g;  # get text between start and stop delimitors with a space
s/ $//                    # remove trailing space

Calling this as:

$ sed -f glean.sed test.html
seconds reqs %reqs Gbytes %bytes
0 10927  0.47% 0.01  0.18%
&lt;= 0.01 1026471 44.59% 0.11  1.81%
0.01-0.02 535390 23.26% 0.06  0.95%
0.02-0.05 93298  4.05% 0.27  4.29%

Then you can use what you like to get the first two fields (as I suggest with cut).

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.