This question already has an answer here:

I would like to "parse" some list of web pages (html), the "parsing" is only about redirecting what's between <title> </title> tags to some specific files. Firstly I did for cycle with wget, secondly came to conclusion (with time of course) that curl is a bit faster in this case.

So it looks like:

for page in $(cat source.txt)
 echo "$(curl -s https://somewebpage/some_sub_page/$page \
 | grep '<title>' -A2 | sed -n '2p')" > tmp/$page/index

And all these subpages contain info I'm interested in on the line after <title> , like here


so I'm just applying common shell tools to reach the result. What's my question. With ~400 pages it takes extremelly a lot of time to load the whole page, then to grep it and so on. Is there some more sofisticated solution to load for e.g. only first 10 lines of html document? I rely on classic bash utilities (curl, wget), but you can recommend what you think fits better in this issue.

UPD: maybe it's not the perfect solution, but adding head after curl decreased script's time in 2 times. (duplicate)

marked as duplicate by DarkHeart, Jeff Schaller, GAD3R, Sparhawk, Kiwy Mar 1 '18 at 15:57

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • I know the fastest way ... but show at least a few actual urls (for my real test case) – RomanPerekhrest Feb 28 '18 at 14:31
  • @RomanPerekhrest it's just my faculty's web, where several links refer to the several classes. So I am parsing classes whole names from each page <title> – Dmytro Osaulenko Feb 28 '18 at 14:56
  • @DmytroOsaulenko, it's just my faculty's web - that wouldn't help – RomanPerekhrest Feb 28 '18 at 15:00
  • @RomanPerekhrest anyway the problem is solved with curl example.com | head – Dmytro Osaulenko Feb 28 '18 at 15:07

Try this using a proper expression :

Command :

saxon-lint --html --xpath '//title/text()' http://domain.tld/path

Check saxon-lint (own project)

You can test (but you need to hide STDERR most of the time (and no https support), because the HTML parser is not that good as what I wrote as command before)

xmllint --html --xpath '//title/text()' http://domain.tld/path 2>/dev/null

Finally :

for page in $(cat source.txt); do
    chosen_command "https://somewebpage/some_sub_page/$page" > "/tmp/$page/index"

You can pipe what you are fetching to this:

wget -4 -qO- {$query} |\
html2text |\
awk '/<title>/,/<\/title>/{print}'
  • Sorry, too slow with this solution – Dmytro Osaulenko Feb 28 '18 at 15:12
  • ... but with common tools ;) – John Goofy Feb 28 '18 at 15:53

Usually the limiting factor with the performance of HTTP is latency, not bandwidth. So I wouldn't expect only loading part of the page to have a significant performance impact. However HTTP does provide a mechanism for doing just this - range requests. And, of course, it's supported by cURL.

However there is no hard requirement for the title to appear near the beginning of the HTML content, only that it appear in the HEAD.

With ~400 pages it takes extremelly a lot of time to load the whole page

As per above, I don't think you're going to gain much here (although if I'm wrong, then you should be looking at implementing a stream based processor rather than your datagram model). If you want to get the results faster, shard the workload and run multiple handler in parallel.

Not the answer you're looking for? Browse other questions tagged or ask your own question.