1

i have been able to use the code below to grep lists of urls from html source before, but for some reason it's not working for this specific example.

grep:

grep -1 box-download shareit1.txt|sed 's/<a/\/n/'|sed 's/href/\/n/'|grep http|cut -d\" -f2>> shareit2.txt

url:

<div class="box-download">
<a data-no-file="0" title="SHAREit free download" href="http://gsf-cf.softonic.com/c98/1a8/173dd01ec9001985d81eb5f2023b03280c/LenovoShareIt-win.exe?SD_used=0&amp;channel=WEB&amp;fdh=no&amp;id_file=69703978&amp;instance=softonic_en&amp;type=PROGRAM&amp;Expires=1444364906&amp;Signature=SdKSfTDHY4dG6HVu2--lqt8lRbGK9S1opIDZiSNwvggAAAXB3hESz1G1Y00rU5iLGY5lai0YOJBXhE4y6gvL4uQvCV4U5jzLDU9TmFTxe4xNDrEmkSC95LyGdGSudQKfrWdD06gBlVrqE49AeeotENtdA3SpkmfQGGd1tnjS138_&amp;Key-Pair-Id=APKAJUA62FNWTI37JTGQ&amp;filename=LenovoShareIt-win.exe" id="download-button" class="button-main-download-xl"
        data-ua="#c,#l,a=Download,downloadType=HostedDownload"
    >
    <strong>Free Download
        <span>Safe download</span>
    </strong>
    <i class="icon-download-alt"></i>
</a>

Appreciate the help.

  • With GNU grep: grep -Poz 'href="\K[^"]*' file.html – Cyrus Oct 8 '15 at 19:04
  • You might find it easier to get what you want by processing the output of lynx -force_html -dump --listonly shareit1.txt. – cas Oct 8 '15 at 23:07
  • @Lewandajo: seems that you have some troubles with this site. Let me help. Near each answer, on the left, there are four items. From top: vote up (^), number, vote down (v) and accept (). You can cast votes for any number of answers. If you think that answer is well-written and helpful, it deserves up vote - use ^. You can also accept answer, but only one for question. is for answer that is the highest quality of all or directly solved your problem. – Mirek Długosz Oct 9 '15 at 12:48
0
sed 's/^[^"]*  *//
     s/" */"\n/2
      /\n/P;D
'    <in >out

That will take turns on printing and eating input lines one double-quoted context at a time. It will probably make your data a lot more grep friendly. As written it doesn't work if quoted contexts can span newlines, but, as I understand it, they're not supposed to in HTML.

Anyway, it does make your sample a lot easier to handle:

class="box-download"
data-no-file="0"
title="SHAREit free download"
href="http://gsf-cf.softonic.com/c98/1a8/173dd01ec9001985d81eb5f2023b03280c/LenovoShareIt-win.exe?SD_used=0&amp;channel=WEB&amp;fdh=no&amp;id_file=69703978&amp;instance=softonic_en&amp;type=PROGRAM&amp;Expires=1444364906&amp;Signature=SdKSfTDHY4dG6HVu2--lqt8lRbGK9S1opIDZiSNwvggAAAXB3hESz1G1Y00rU5iLGY5lai0YOJBXhE4y6gvL4uQvCV4U5jzLDU9TmFTxe4xNDrEmkSC95LyGdGSudQKfrWdD06gBlVrqE49AeeotENtdA3SpkmfQGGd1tnjS138_&amp;Key-Pair-Id=APKAJUA62FNWTI37JTGQ&amp;filename=LenovoShareIt-win.exe"
id="download-button"
class="button-main-download-xl"
data-ua="#c,#l,a=Download,downloadType=HostedDownload"
class="icon-download-alt"
1

With this (fixed) file.html:

<html>
  <div class="box-download">
    <a data-no-file="0" title="SHAREit free download" href="http://gsf-cf.softonic.com/c98/1a8/173dd01ec9001985d81eb5f2023b03280c/LenovoShareIt-win.exe?SD_used=0&amp;channel=WEB&amp;fdh=no&amp;id_file=69703978&amp;instance=softonic_en&amp;type=PROGRAM&amp;Expires=1444364906&amp;Signature=SdKSfTDHY4dG6HVu2--lqt8lRbGK9S1opIDZiSNwvggAAAXB3hESz1G1Y00rU5iLGY5lai0YOJBXhE4y6gvL4uQvCV4U5jzLDU9TmFTxe4xNDrEmkSC95LyGdGSudQKfrWdD06gBlVrqE49AeeotENtdA3SpkmfQGGd1tnjS138_&amp;Key-Pair-Id=APKAJUA62FNWTI37JTGQ&amp;filename=LenovoShareIt-win.exe" id="download-button" class="button-main-download-xl" data-ua="#c,#l,a=Download,downloadType=HostedDownload">
      <strong>Free Download<span>Safe download</span></strong>
      <i class="icon-download-alt"></i>
    </a>
  </div>
</html>

Command:

xmlstarlet sel -t -v "//html/div/a/@href" file.html

Output:

http://gsf-cf.softonic.com/c98/1a8/173dd01ec9001985d81eb5f2023b03280c/LenovoShareIt-win.exe?SD_used=0&channel=WEB&fdh=no&id_file=69703978&instance=softonic_en&type=PROGRAM&Expires=1444364906&Signature=SdKSfTDHY4dG6HVu2--lqt8lRbGK9S1opIDZiSNwvggAAAXB3hESz1G1Y00rU5iLGY5lai0YOJBXhE4y6gvL4uQvCV4U5jzLDU9TmFTxe4xNDrEmkSC95LyGdGSudQKfrWdD06gBlVrqE49AeeotENtdA3SpkmfQGGd1tnjS138_&Key-Pair-Id=APKAJUA62FNWTI37JTGQ&filename=LenovoShareIt-win.exe
  • Using XPath is the only correct approach, but I'm afraid that requirement of "fixing" HTML content will not cut it in. Adding <html> and </html> around content provided by OP is doable, but it's probably not source of entire page. So you have to first extract that <div>. You probably can't use xmlstarlet, because some other parts might require "fixing". On the other hand, simple grep will not handle variable number of lines that <div> might span. This is kind of chicken-and-egg problem and it shows that shell toolset is not very suitable for handling HTML content. – Mirek Długosz Oct 8 '15 at 18:43
  • Thanks @MirosławZalewski As you said, i am having a problem with shell toolset, a better way of handling html seems to be DOMXPath. – Lewandajo Oct 9 '15 at 9:23
0

It can't work, because you have the slashes backwards (unless there is StackExchange parser messing things up). You have figured out that in order to select attribute content, you will put it into new line and cut from there. But newline character is \n, not \/n. The second one is "/n, ensuring that forward slash is treated literally".

So this code, after the fix, will work:

grep -1 box-download shareit1.txt|sed 's:<a:\n:'|sed 's/href/\n/'|grep http |cut -d\" -f 2

Although you could cut down the number of pipes and invocations of external programs by relying solely on sed:

sed -n -e '/http/ s:.*href="\([^"]*\)".*:\1: p' shareit1.txt

On side note, all these approaches are wrong anyway. The correct way is using DOM/XPath. The problem is that there is no DOM parser for the shell (that I am aware of) and common shell XML parsers that allow to specify XPath expressions expect file to be valid XML - which website content almost never is. In other programming languages, like Python, ruby or perl (to mention the most common ones for system admins), there are libraries that would turn messy HTML file into something that resembles valid XML file and what could be fed into XML parser.

  • Thanks again, this works a treat. I need to use the extracted url to download the program, using wget variables as I will be downloading lots of programs. Will also need the filename it saves as to be variable, any suggestions on how to do this? – Lewandajo Oct 9 '15 at 10:58
  • If server is configured properly, wget should save data under correct file name automatically. If it does not, try --trust-server-names and/or --content-disposition options. In this particular example, you can extract file name using \/\([^/?]*?\)\? (for one between last slash and question mark) or .*=\(.*\)$ (for one after last equal sign). If URL is already in variable $URL, use something like that: echo "$URL" |sed -e 's:.*=\(.*\)$:\1:' (untested, as I'm not on Linux right now). – Mirek Długosz Oct 9 '15 at 11:15
  • Also, don't forget to click up arrow near answers that you find helpful and accept answer that you... well, accept. – Mirek Długosz Oct 9 '15 at 11:15
0

You can use lynx -dump --list-only to extract a list of URLs from the html input.

If you don't already have lynx installed, you'll have to install it using whatever mechanism is normal for your linux or unix system (e.g. apt-get, yum, make install, etc)

e.g.

$ lynx -force_html -dump --listonly shareit1.txt | awk '/^ *[0-9]+\. / { print $2}'
http://gsf-cf.softonic.com/c98/1a8/173dd01ec9001985d81eb5f2023b03280c/LenovoShareIt-win.exe?SD_used=0&channel=WEB&fdh=no&id_file=69703978&instance=softonic_en&type=PROGRAM&Expires=1444364906&Signature=SdKSfTDHY4dG6HVu2--lqt8lRbGK9S1opIDZiSNwvggAAAXB3hESz1G1Y00rU5iLGY5lai0YOJBXhE4y6gvL4uQvCV4U5jzLDU9TmFTxe4xNDrEmkSC95LyGdGSudQKfrWdD06gBlVrqE49AeeotENtdA3SpkmfQGGd1tnjS138_&Key-Pair-Id=APKAJUA62FNWTI37JTGQ&filename=LenovoShareIt-win.exe
0

You can use pup; it's an amazing tool.

Here, see how I download a file and then add particular links from inside of the HTML to a file.

wget http://domain.com -O file.html && pup 'a.className[href] attr{href}' < file.html > links.md

All the links on Domain.com with class name className will be downloaded and saved into a file called links.md.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.