I want a command line program that prints the title of a website. For e.g.:
Alan:~ titlefetcher http://www.youtube.com/watch?v=Dd7dQh8u4Hc
Why Are Bad Words Bad?
You give it the url and it prints out the Title.
You can pipe it to GNU
To remove the
To point out some of the limitations:
There is no standard/portable command to do HTTP queries. A few decades ago, I would have recommended
HTTP protocol and redirection handling
The request sent by wget/w3m/GET... is going to be different from the one sent by firefox. If that's an issue, you can alter
The most important ones here in this regard are:
Here, out of laziness, we have
That way, awk stops reading after the first
Parsing of the HTML
That will work for most HTML pages that have a
By contrast coffeeMug's solution will parse the HTML page as XML and return the corresponding value for
Both my solution and coffeeMug's will fail for a variety of corner cases, sometimes the same, sometimes not.
For instance, mine will fail on:
While his will fail on:
(valid html, not xml) or:
(incorrect html, but still found out there and supported by most browsers)
interpretation of the code inside the tags.
That solution outputs the raw text between
Which is taken care of by GNU
But a web client is also meant to do more transformations on that code when displaying the title (like condense some of the blanks, remove the leading and trailing ones). However it's unlikely that there'd be a need for that. So, as in the other cases, it's up to you do decide whether it's worth the effort.
Before UTF-8, iso8859-1 used to be the preferred charset on the web for non-ASCII characters though strictly speaking they had to be written as
So, that means that out there, you'll find
If your system charset is utf-8, chances are it's going to be alright most of the time as that tends to be the default charset used out there nowadays.
But if you want to cover for other charsets, once again, it would have to be taken care of.
It should also be noted that this solution won't work at all for UTF-16 or UTF-32 encoded pages.
To sum up
Ideally, what you need here, is a real web browser to give you the information. That is, you need something to do the HTTP request with the proper parameters, intepret the HTTP response correctly, fully interpret the HTML code as a browser would, and return the title.
As I don't think that can be done on the command line with the browsers I know (though see now this trick with
While using regexps to parse HTML is often frowned upon, here is a typical case where it's good enough for the task (IMO).
You can also try
You can install
STDERR redirection is to avoid the
In order to get rid of "- YouTube", pipe the output of the above command to
You can also use
The pattern to
More complex situations
And a sample run:
The above finds the case insensitive string
A real HTML/XML Parser - using Ruby
At some point regex will fail in solving this type of problem. If that occurs then you'll likely want to use a real HTML/XML parser. One such parser is Nokogiri. It's available in Ruby as a Gem and can be used like so:
The above is parsing the data that comes via the
A real HTML/XML Parser - using Perl
You can also do something similar with Perl and the HTML::TreeBuilder::XPath module.
You can then run this script like so:
Using simple regex to parse HTML is naive. E.g. with newlines and ignoring special character encoding specified in the file. Do the right thing and really parse the page using any of the other real parsers mentioned in the other answers or use the following one liner:
(The above includes a Unicode character).
BeautifulSoup handles a lot of incorrect HTML (e.g. missing closing tags) as well, that would completely throw of simplistic regexing. You can install it in a standard python using:
or if you don't have
Some operating systems like Debian/Ubuntu also have it packaged (
It seems to be possible with
Because that's a real life web browser, it doesn't suffer from many of the limitations I mention in my other answer.
Here, we're using the fact that
Above, we're giving a configuration file (as a pipe) that defines a lynx "printer" called
Then we use
Maybe it's "cheating" but one option is pup, a command line HTML parser.
Here are two ways to do it:
and another way using the