1

I know there are command-line web browsers like w3m lynx etc. But I am looking for a simpler tool to just print search queries. Somewhat like curl does, but more nicely formatted.

Like if I run https://www.youtube.com/results?search_query=linux

I would get something like:

Top 3 Best Desktop Linux Distros | 2019 Edition
https://www.youtube.com/watch?v=vZ7nI2im4Yg
_________________________________________________
The Next Ubuntu? - Clear Linux First Impressions
https://www.youtube.com/watch?v=UuCxCLFDfwM
_________________________________________________
The Next Ubuntu? - Clear Linux First Impressions
https://www.youtube.com/watch?v=FUGd99GRAfo
_________________________________________________

in simpler words just Title and Link


EDIT: with thanks to the comments I did some further research on web-scraping and found this code using wget and two applications from the html-xml-utils package in combination with lynx:

   link="https://www.youtube.com/results?search_query=linux"

   wget "$link" -O- |
   hxnormalize -x |
   hxselect -i "div." |  #in this line I'm missing the proper pattern
   lynx -stdin -dump

if I inspect the title displayed in a youtube.com search I get:

<div id="title-wrapper" class="style-scope ytd-video-renderer">
  <h3 class="title-and-badge style-scope ytd-video-renderer">
    <ytd-badge-supported-renderer class="style-scope ytd-video-renderer" disable-upgrade="" hidden="">

    <dom-repeat id="repeat" as="badge" class="style-scope ytd-badge-supported-renderer" style="display: none;"><template></template></dom-repeat>
    </ytd-badge-supported-renderer>
    <a id="video-title" class="yt-simple-endpoint style-scope ytd-video-renderer" title="What is Linux?" href="/watch?v=zA3vmx0GaO8" aria-l$
                What is Linux?
    </a>
  </h3>
<div id="menu" class="style-scope ytd-video-renderer"></div>
</div>

I do not get what pattern to use in hxselect in order to grep the title and link.

  • 5
    Sounds like you want to extract all external links of a web page. – Martin May 27 at 23:02
  • 5
    There's no general way to do this that automatically leads to "nicely formatted output", you have to analyze the web page on a case-by-case basis. This is also known as "web scraping". There are command line tools like sgrep that can assist in parsing the web page. – dirkt May 28 at 1:44
  • 2
    I can also suggest python in combination with beautifulsoup for web scraping. – Panki May 28 at 7:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.