I want a command line program that prints the title of a website. For e.g.:
Alan:~ titlefetcher http://www.youtube.com/watch?v=Dd7dQh8u4Hc
should give:
Why Are Bad Words Bad?
You give it the url and it prints out the Title.
-
2When I download that title I get: "Why Are Bad Words Bad? - Youtube", do you want the "- Youtube" truncated too?– slm ♦Dec 1, 2013 at 17:28
Add a comment
|
12 Answers
wget -qO- 'http://www.youtube.com/watch?v=Dd7dQh8u4Hc' |
perl -l -0777 -ne 'print $1 if /<title.*?>\s*(.*?)\s*<\/title/si'
You can pipe it to GNU recode if there are things like < in it:
wget -qO- 'http://www.youtube.com/watch?v=Dd7dQh8u4Hc' |
perl -l -0777 -ne 'print $1 if /<title.*?>\s*(.*?)\s*<\/title/si' |
recode html..
To remove the - youtube part:
wget -qO- 'http://www.youtube.com/watch?v=Dd7dQh8u4Hc' |
perl -l -0777 -ne 'print $1 if /<title.*?>\s*(.*?)(?: - youtube)?\s*<\/title/si'
To point out some of the limitations:
portability
There is no standard/portable command to do HTTP queries. A few decades ago, I would have recommended lynx -source instead here. But nowadays, wget is more portable as it can be found by default on most GNU systems (including most Linux-based desktop/laptop operating systems). Other fairly portables ones include the GET command that comes with perl's libwww that is often installed, lynx -source, and to a lesser extent curl. Other common ones include links -source, elinks -source, w3m -dump_source, lftp -c cat...
HTTP protocol and redirection handling
wget may not get the same page as the one that for instance firefox would display. The reason being that HTTP servers may choose to send a different page based on the information provided in the request sent by the client.
The request sent by wget/w3m/GET... is going to be different from the one sent by firefox. If that's an issue, you can alter wget behaviour to change the way it sends the request though with options.
The most important ones here in this regard are:
AcceptandAccept-language: that tells the server in which language and charset the client would like to get the response in.wgetdoesn't send any by default so the server will typically send with its default settings.firefoxon the other end is likely configured to request your language.User-Agent: that identifies the client application to the server. Some sites send different content based on the client (though that's mostly for differences between javascript language interpretations) and may refuse to serve you if you're using a robot-type user agent likewget.Cookie: if you've visited this site before, your browser may have permanent cookies for it.wgetwill not.
wget will follow the redirections when they are done at the HTTP protocol level, but since it doesn't look at the content of the page, not the ones done by javascript or things like <meta http-equiv="refresh" content="0; url=http://example.com/">.
Performance/Efficiency
Here, out of laziness, we have perl read the whole content in memory before starting to look for the <title> tag. Given that the title is found in the <head> section that is in the first few bytes of the file, that's not optimal. A better approach, if GNU awk is available on your system could be:
wget -qO- 'http://www.youtube.com/watch?v=Dd7dQh8u4Hc' |
gawk -v IGNORECASE=1 -v RS='</title' 'RT{gsub(/.*<title[^>]*>/,"");print;exit}'
That way, awk stops reading after the first </title, and by exiting, causes wget to stop downloading.
Parsing of the HTML
Here, wget writes the page as it downloads it. At the same time, perl, slurps its output (-0777 -n) whole in memory and then prints the HTML code that is found between the first occurrences of <title...> and </title.
That will work for most HTML pages that have a <title> tag, but there are cases where it won't work.
By contrast coffeeMug's solution will parse the HTML page as XML and return the corresponding value for title. It is more correct if the page is guaranteed to be valid XML. However, HTML is not required to be valid XML (older versions of the language were not), and because most browsers out there are lenient and will accept incorrect HTML code, there's even a lot of incorrect HTML code out there.
Both my solution and coffeeMug's will fail for a variety of corner cases, sometimes the same, sometimes not.
For instance, mine will fail on:
<html><head foo="<title>"><title>blah</title></head></html>
or:
<!-- <title>old</title> --><title>new</title>
While his will fail on:
<TITLE>foo</TITLE>
(valid html, not xml) or:
or:
<title>...</title>
...
<script>a='<title>'; b='</title>';</script>
(again, valid html, missing <
-
8
-
1You'll likely want to add
-itohxselecttoo for the same reasons that manatwork mentioned in my A, otherwise it won't match<TITLE>.– slm ♦Dec 1, 2013 at 18:09 -
1I haven't found what to do with the msg. about normalizing the output. No such switch on
hxselect.– slm ♦Dec 1, 2013 at 18:10 -
3
-
1I was having issues with hxselect completely refusing to work on some poorly-structured HTML, so I had to pipe it through hxclean first (e.g.
wget -qO- 'http://www.youtube.com/watch?v=Dd7dQh8u4Hc' | hxclean | hxselect -s '\n' -c 'title' 2>/dev/null). Sep 22, 2019 at 5:52
You can also use curl and grep to do this. You'll need to enlist the use of PCRE (Perl Compatible Regular Expressions) in grep to get the look behind and look ahead facilities so that we can find the <title>...</title> tags.
Example
$ curl 'http://www.youtube.com/watch?v=Dd7dQh8u4Hc' -so - | \
grep -iPo '(?<=<title>)(.*)(?=</title>)'
Why Are Bad Words Bad? - YouTube
Details
The curl switches:
-s= silent-o -= send output to STDOUT
The grep switches:
-i= case insensitivity-o= Return only the portion that matches-P= PCRE mode
The pattern to grep:
(?<=<title>)= look for a string that starts with this to the left of it(?=</title>)= look for a string that ends with this to the right of it(.*)= everything in between<title>..</title>.
More complex situations
If <title>...</titie> spans multiple lines, then the above won't find it. You can mitigate this situation by using tr, to delete any \n characters, i.e. tr -d '\n'.
Example
Sample file.
$ cat multi-line.html
<html>
<title>
this is a \n title
</TITLE>
<body>
<p>this is a \n title</p>
</body>
</html>
And a sample run:
$ curl 'http://www.jake8us.org/~sam/multi-line.html' -so - | \
tr -d '\n' | \
grep -iPo '(?<=<title>)(.*)(?=</title>)'
this is a \n title
lang=...
If the <title> is set like this, <title lang="en"> then you'll need to remove this prior to greping it. The tool sed can be used to do this:
$ curl 'http://www.jake8us.org/~sam/multi-line.html' -so - | \
tr -d '\n' | \
sed 's/ lang="\w+"//gi' | \
grep -iPo '(?<=<title>)(.*)(?=</title>)'
this is a \n title
The above finds the case insensitive string lang= followed by a word sequence (\w+). It is then stripped out.
A real HTML/XML Parser - using Ruby
At some point regex will fail in solving this type of problem. If that occurs then you'll likely want to use a real HTML/XML parser. One such parser is Nokogiri. It's available in Ruby as a Gem and can be used like so:
$ curl 'http://www.jake8us.org/~sam/multi-line.html' -so - | \
ruby -rnokogiri -e \
'puts Nokogiri::HTML(readlines.join).xpath("//title").map { |e| e.content }'
this is a \n title
The above is parsing the data that comes via the curl as HTML (Nokogiri::HTML). The method xpath then looks for nodes (tags) in the HTML that are leaf nodes, (//) with the name title. For each found we want to return its content (e.content). The puts then prints them out.
A real HTML/XML Parser - using Perl
You can also do something similar with Perl and the HTML::TreeBuilder::XPath module.
$ cat title_getter.pl
#!/usr/bin/perl
use HTML::TreeBuilder::XPath;
$tree = HTML::TreeBuilder::XPath->new_from_url($ARGV[0]);
($title = $tree->findvalue('//title')) =~ s/^\s+//;
print $title . "\n";
You can then run this script like so:
$ ./title_getter.pl http://www.jake8us.org/~sam/multi-line.html
this is a \n title
-
1
-
@coffeMug - thanks, seemed more straight forward than the other 2 methods.– slm ♦Dec 1, 2013 at 17:47
-
5Parsing HTML with regular expressions is not so simple. Tags written as “<TITLE>”, “<title lang=en>”, “<title\n>” will not be matched by your expression. Even bigger problem, neither “<title>\noops\n</title>” will be. Dec 1, 2013 at 18:00
-
5Attempting to parse html using regex tends to be frowned upon round here.– user3490Dec 1, 2013 at 21:50
-
1@slm,
<title>Unix\nLinux</title>is meant to beUnix Linux, notUnixLinux. Dec 2, 2013 at 13:10
Using simple regex to parse HTML is naive. E.g. with newlines and ignoring special character encoding specified in the file. Do the right thing and really parse the page using any of the other real parsers mentioned in the other answers or use the following one liner:
python -c "import bs4, urllib2; print bs4.BeautifulSoup(urllib2.urlopen('http://www.crummy.com/software/BeautifulSoup/bs4/doc/')).title.text"
(The above includes a Unicode character).
BeautifulSoup handles a lot of incorrect HTML (e.g. missing closing tags) as well, that would completely throw of simplistic regexing. You can install it in a standard python using:
pip install beautifulsoup4
or if you don't have pip, with
easy_install beautifulsoup4
Some operating systems like Debian/Ubuntu also have it packaged (python-bs4 package on Debian/Ubuntu).
-
2
bs4is not in the python standard library. You have to install it usingeasy_install beautfulsoup4(noteasyinstall bs4).– AnthonDec 2, 2013 at 8:07 -
-
Prints the
SyntaxError: Missing parentheses in call to 'print'. Did you mean print(...)?error on Python 3.11.3 / beautifulsoup4-4.12.2-1 Jun 30 at 7:47 -
A more recent answer may work, I tested with
python -c "import bs4, urllib2; print bs4.BeautifulSoup(urllib2.urlopen('https://example.com')).title.text"Jun 30 at 7:49
Simple way:
curl -s example.com | grep -o "<title>[^<]*" | tail -c+8
Few alternatives:
curl -s example.com | grep -o "<title>[^<]*" | cut -d'>' -f2-
wget -qO- example.com | grep -o "<title>[^<]*" | sed -e 's/<[^>]*>//g'
-
2
-
All three sadly struggle with characters such as
&and", displayed as&and"Jun 10, 2022 at 11:06
Maybe it's "cheating" but one option is pup, a command line HTML parser.
Here are two ways to do it:
Using the meta field with property="og:title attribute
$ wget -q 'http://www.youtube.com/watch?v=Dd7dQh8u4Hc' -O - | \
> pup 'meta[property=og:title] attr{content}'
Why Are Bad Words Bad?
and another way using the title field directly (and then lopping off the - YouTube string at the end).
$ wget -q 'http://www.youtube.com/watch?v=Dd7dQh8u4Hc' -O - | \
> pup 'title text{}' | sed 's/ - YouTube$//'
Why Are Bad Words Bad?
-
To avoid character entities, users may wish to use pup's
--plainoption.– peakJun 21, 2017 at 7:57
It seems to be possible with lynx using this trick:
lynx 3>&1 > /dev/null -nopause -noprint -accept_all_cookies \
-cmd_script /dev/stdin<<'EOF' 'http://www.youtube.com/watch?v=Dd7dQh8u4Hc'
set PRINTER=P:printf '%0s\\n' "$LYNX_PRINT_TITLE">&3:TRUE
key p
key Select key
key ^J
exit
EOF
Because that's a real life web browser, it doesn't suffer from many of the limitations I mention in my other answer.
Here, we're using the fact that lynx sets the $LYNX_PRINT_TITLE environment variable to the title of the current page when printing the page.
Above, we use lynx scripting facility (with the script passed on stdin via a heredocument) to:
- define a lynx "printer" called
Pthat just outputs the content of that variable to file descriptor3(that file descriptor is redirected tolynx's stdout with3>&1while lynx stdout is itself redirected to /dev/null). - simulate the user pressing p, and the End (aka Select), and Enter (
^J).
answered Dec 2, 2013 at 11:37
A python3 + beautifulsoup example might be
python3 -c "import bs4, requests; print(bs4.BeautifulSoup(requests.get('http://www.crummy.com/software/BeautifulSoup/bs4/doc/').content).title.text)"
I liked the idea of Stéphane Chazelas to use Lynx and LYNX_PRINT_TITLE, but that script didn't work for me under Ubuntu 14.04.5.
I have made a simplified version of it by using running Lynx and using files pre-configured in advance.
Add the following line to /etc/lynx-cur/lynx.cfg (or wherever your lynx.cfg resides):
PRINTER:P:printenv LYNX_PRINT_TITLE>/home/account/title.txt:TRUE:1000
This line instructs to save title, while printing, to "/home/account/title.txt" - you may choose any file name you wish. You request VERY large pages, increase the above value from "1000" to any number of lines per page you want, otherwise Lynx will make additional prompt "when printing document containing very large number of pages".
Then create the /home/account/lynx-script.txt file with the following contents:
key p
key Select key
key ^J
exit
Then run Lynx using the following command-line options:
lynx -term=vt100 -display_charset=utf-8 -nopause -noprint -accept_all_cookies -cmd_script=/home/account/lynx-script.txt "http://www.youtube.com/watch?v=Dd7dQh8u4Hc" >/dev/nul
Upon completion of this command, the file /home/account/title.txt will be created with the title of your page.
Long story short, here is a PHP function that returns a page title based on the given URL, or false in case of error.
function GetUrlTitle($url)
{
$title_file_name = "/home/account/title.txt";
if (file_exists($title_file_name)) unlink($title_file_name); // delete the file if exists
$cmd = '/usr/bin/lynx -cfg=/etc/lynx-cur/lynx.cfg -term=vt100 -display_charset=utf-8 -nopause -noprint -accept_all_cookies -cmd_script=/home/account/lynx-script.txt "'.$url.'"';
exec($cmd, $output, $retval);
if (file_exists($title_file_name))
{
$title = file_get_contents($title_file_name);
unlink($title_file_name); // delete the file after reading
return $title;
} else
{
return false;
}
}
print GetUrlTitle("http://www.youtube.com/watch?v=Dd7dQh8u4Hc");
Using nokogiri, one can use a simple CSS-based query to extract the inner text of the tag:
$ nokogiri -e 'puts $_.at_css("title").content'
Why Are Bad Words Bad? - YouTube
Similarly, to extract the value of the "content" attribute of the tag:
$ nokogiri -e 'puts $_.at_css("meta[name=title]").attr("content")'
Why Are Bad Words Bad?
Using htmlq:
curl --silent "https://www.youtube.com/watch?v=Dd7dQh8u4Hc" | htmlq --text title
If you do not have it installed, htmlq can be built with cargo:
cargo install htmlq
Using xidel:
$ xidel -s http://www.youtube.com/watch?v=Dd7dQh8u4Hc --css title
Why Are Bad Words Bad? - YouTube
If necessary, apt install xidel or similar.
