This question already has an answer here:
I would like to "parse" some list of web pages (html), the "parsing" is only about redirecting what's between
<title> </title> tags to some specific files.
Firstly I did
for cycle with
wget, secondly came to conclusion (with
time of course) that
curl is a bit faster in this case.
So it looks like:
for page in $(cat source.txt) do echo "$(curl -s https://somewebpage/some_sub_page/$page \ | grep '<title>' -A2 | sed -n '2p')" > tmp/$page/index done
And all these subpages contain info I'm interested in on the line after
<title> ... </title>
so I'm just applying common shell tools to reach the result.
What's my question. With ~400 pages it takes extremelly a lot of time to load the whole page, then to
grep it and so on. Is there some more sofisticated solution to load for e.g. only first 10 lines of html document? I rely on classic bash utilities (curl, wget), but you can recommend what you think fits better in this issue.
UPD: maybe it's not the perfect solution, but adding
curl decreased script's
time in 2 times. (duplicate)