6

Do I first need to run html in the wild through something like jsoup? Not to make it valid in a human sense, might turn it to gibberish, but at least so that xmlstarlet can process the file?

Preferably looking for a CLI which can be installed and used like so:

massage foo.html > bar.xhtml

or at least something along those lines.

Use case:

thufir@doge:~/.html$ 
thufir@doge:~/.html$ curl http://int.soccerway.com/  > soccer.html
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  188k    0  188k    0     0   313k      0 --:--:-- --:--:-- --:--:--  313k
thufir@doge:~/.html$ 
thufir@doge:~/.html$ xmlstarlet sel -t -v "/html/body/table/tr/td[1]" -n soccer.html 
soccer.html:70.13: xmlParseEntityRef: no name
if (this.$ && this.$.fn && this.$.fn.jquery) {
            ^
soccer.html:70.14: xmlParseEntityRef: no name
if (this.$ && this.$.fn && this.$.fn.jquery) {
             ^
soccer.html:70.26: xmlParseEntityRef: no name
if (this.$ && this.$.fn && this.$.fn.jquery) {
                         ^
soccer.html:70.27: xmlParseEntityRef: no name
if (this.$ && this.$.fn && this.$.fn.jquery) {
                          ^
soccer.html:198.8: Opening and ending tag mismatch: link line 27 and head
</head>
       ^
soccer.html:209.45: EntityRef: expecting ';'
  j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src=
                                            ^
soccer.html:223.40: xmlParseEntityRef: no name
      if (typeof(e.data) === 'string' && (e.data.indexOf('onEplayerVideoStarted'
                                       ^

Ideally would run htmlstarlet directly against the URL but there seems no such provision.

there is a fo option to format, but I wasn't able to get different results than above.

3

If you just want the table's data cells, it is possible using xmlstarlet fo followed by xmlstarlet sel. The main issue you are having, is with the xpath. If you add a couple of "wildcard" elements (//), you get the desired result:

# fetch url and send to standard out (-s)
curl -sL http://int.soccerway.com/                          |

# interpret input as html (-H) and try to recover as much as possible (-R)
xmlstarlet fo  -H -R                           2> /dev/null |

# use the following xpath expression and return the value (-t -v), 
# also add a newline after the result (-n)
xmlstarlet sel -t -v '//table//tr//h3/span' -n 2> /dev/null |

# only show the first 10 values
head -n10

Output:

World - Friendlies
Argentina - Prim B Nacional
Australia - National Premier Leagues
Australia - NPL Youth League
Bangladesh - Premier League
Belarus - Premier League
Benin - Championnat National
Brazil - Serie A
Brazil - Serie D
Brazil - Copa Paulista
  • bonus points for explaining all those flags and span ;) – Thufir Aug 1 '17 at 0:38
  • 1
    @Thufir: I clarified the text a bit and added some explanation as comments to the pipeline – Thor Aug 1 '17 at 10:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.