Trying to grep url from html source in .txt file using sed

Question

i have been able to use the code below to grep lists of urls from html source before, but for some reason it's not working for this specific example.

grep:

grep -1 box-download shareit1.txt|sed 's/<a/\/n/'|sed 's/href/\/n/'|grep http|cut -d\" -f2>> shareit2.txt

url:

<div class="box-download">
<a data-no-file="0" title="SHAREit free download" href="http://gsf-cf.softonic.com/c98/1a8/173dd01ec9001985d81eb5f2023b03280c/LenovoShareIt-win.exe?SD_used=0&amp;channel=WEB&amp;fdh=no&amp;id_file=69703978&amp;instance=softonic_en&amp;type=PROGRAM&amp;Expires=1444364906&amp;Signature=SdKSfTDHY4dG6HVu2--lqt8lRbGK9S1opIDZiSNwvggAAAXB3hESz1G1Y00rU5iLGY5lai0YOJBXhE4y6gvL4uQvCV4U5jzLDU9TmFTxe4xNDrEmkSC95LyGdGSudQKfrWdD06gBlVrqE49AeeotENtdA3SpkmfQGGd1tnjS138_&amp;Key-Pair-Id=APKAJUA62FNWTI37JTGQ&amp;filename=LenovoShareIt-win.exe" id="download-button" class="button-main-download-xl"
        data-ua="#c,#l,a=Download,downloadType=HostedDownload"
    >
    <strong>Free Download
        <span>Safe download</span>
    </strong>
    <i class="icon-download-alt"></i>
</a>

Appreciate the help.

You might find it easier to get what you want by processing the output of lynx -force_html -dump --listonly shareit1.txt. — cas, Oct 8, 2015 at 23:07
@Lewandajo: seems that you have some troubles with this site. Let me help. Near each answer, on the left, there are four items. From top: vote up (^), number, vote down (v) and accept (✓). You can cast votes for any number of answers. If you think that answer is well-written and helpful, it deserves up vote - use ^. You can also accept answer, but only one for question. ✓ is for answer that is the highest quality of all or directly solved your problem. — Mirek Długosz, Oct 9, 2015 at 12:48

Cyrus · Accepted Answer · 2015-10-08 18:10:02Z

With this (fixed) file.html:

<html>
  <div class="box-download">
    <a data-no-file="0" title="SHAREit free download" href="http://gsf-cf.softonic.com/c98/1a8/173dd01ec9001985d81eb5f2023b03280c/LenovoShareIt-win.exe?SD_used=0&amp;channel=WEB&amp;fdh=no&amp;id_file=69703978&amp;instance=softonic_en&amp;type=PROGRAM&amp;Expires=1444364906&amp;Signature=SdKSfTDHY4dG6HVu2--lqt8lRbGK9S1opIDZiSNwvggAAAXB3hESz1G1Y00rU5iLGY5lai0YOJBXhE4y6gvL4uQvCV4U5jzLDU9TmFTxe4xNDrEmkSC95LyGdGSudQKfrWdD06gBlVrqE49AeeotENtdA3SpkmfQGGd1tnjS138_&amp;Key-Pair-Id=APKAJUA62FNWTI37JTGQ&amp;filename=LenovoShareIt-win.exe" id="download-button" class="button-main-download-xl" data-ua="#c,#l,a=Download,downloadType=HostedDownload">
      <strong>Free Download<span>Safe download</span></strong>
      <i class="icon-download-alt"></i>
    </a>
  </div>
</html>

Command:

xmlstarlet sel -t -v "//html/div/a/@href" file.html

Output:

http://gsf-cf.softonic.com/c98/1a8/173dd01ec9001985d81eb5f2023b03280c/LenovoShareIt-win.exe?SD_used=0&channel=WEB&fdh=no&id_file=69703978&instance=softonic_en&type=PROGRAM&Expires=1444364906&Signature=SdKSfTDHY4dG6HVu2--lqt8lRbGK9S1opIDZiSNwvggAAAXB3hESz1G1Y00rU5iLGY5lai0YOJBXhE4y6gvL4uQvCV4U5jzLDU9TmFTxe4xNDrEmkSC95LyGdGSudQKfrWdD06gBlVrqE49AeeotENtdA3SpkmfQGGd1tnjS138_&Key-Pair-Id=APKAJUA62FNWTI37JTGQ&filename=LenovoShareIt-win.exe

Using XPath is the only correct approach, but I'm afraid that requirement of "fixing" HTML content will not cut it in. Adding <html> and </html> around content provided by OP is doable, but it's probably not source of entire page. So you have to first extract that <div>. You probably can't use xmlstarlet, because some other parts might require "fixing". On the other hand, simple grep will not handle variable number of lines that <div> might span. This is kind of chicken-and-egg problem and it shows that shell toolset is not very suitable for handling HTML content. — Mirek Długosz, Oct 8, 2015 at 18:43
Thanks @MirosławZalewski As you said, i am having a problem with shell toolset, a better way of handling html seems to be DOMXPath. — Lewandajo, Oct 9, 2015 at 9:23

mikeserv · Accepted Answer · 2015-10-08 18:17:35Z

sed 's/^[^"]*  *//
     s/" */"\n/2
      /\n/P;D
'    <in >out

That will take turns on printing and eating input lines one double-quoted context at a time. It will probably make your data a lot more grep friendly. As written it doesn't work if quoted contexts can span newlines, but, as I understand it, they're not supposed to in HTML.

Anyway, it does make your sample a lot easier to handle:

class="box-download"
data-no-file="0"
title="SHAREit free download"
href="http://gsf-cf.softonic.com/c98/1a8/173dd01ec9001985d81eb5f2023b03280c/LenovoShareIt-win.exe?SD_used=0&amp;channel=WEB&amp;fdh=no&amp;id_file=69703978&amp;instance=softonic_en&amp;type=PROGRAM&amp;Expires=1444364906&amp;Signature=SdKSfTDHY4dG6HVu2--lqt8lRbGK9S1opIDZiSNwvggAAAXB3hESz1G1Y00rU5iLGY5lai0YOJBXhE4y6gvL4uQvCV4U5jzLDU9TmFTxe4xNDrEmkSC95LyGdGSudQKfrWdD06gBlVrqE49AeeotENtdA3SpkmfQGGd1tnjS138_&amp;Key-Pair-Id=APKAJUA62FNWTI37JTGQ&amp;filename=LenovoShareIt-win.exe"
id="download-button"
class="button-main-download-xl"
data-ua="#c,#l,a=Download,downloadType=HostedDownload"
class="icon-download-alt"

Mirek Długosz · Accepted Answer · 2015-10-08 18:13:59Z

0

It can't work, because you have the slashes backwards (unless there is StackExchange parser messing things up). You have figured out that in order to select attribute content, you will put it into new line and cut from there. But newline character is \n, not \/n. The second one is "/n, ensuring that forward slash is treated literally".

So this code, after the fix, will work:

grep -1 box-download shareit1.txt|sed 's:<a:\n:'|sed 's/href/\n/'|grep http |cut -d\" -f 2

Although you could cut down the number of pipes and invocations of external programs by relying solely on sed:

sed -n -e '/http/ s:.*href="\([^"]*\)".*:\1: p' shareit1.txt

On side note, all these approaches are wrong anyway. The correct way is using DOM/XPath. The problem is that there is no DOM parser for the shell (that I am aware of) and common shell XML parsers that allow to specify XPath expressions expect file to be valid XML - which website content almost never is. In other programming languages, like Python, ruby or perl (to mention the most common ones for system admins), there are libraries that would turn messy HTML file into something that resembles valid XML file and what could be fed into XML parser.

edited Oct 8, 2015 at 18:13

answered Oct 8, 2015 at 18:02

Mirek Długosz

9731 gold badge7 silver badges9 bronze badges

Thanks again, this works a treat. I need to use the extracted url to download the program, using wget variables as I will be downloading lots of programs. Will also need the filename it saves as to be variable, any suggestions on how to do this?
– Lewandajo
Oct 9, 2015 at 10:58
If server is configured properly, wget should save data under correct file name automatically. If it does not, try --trust-server-names and/or --content-disposition options. In this particular example, you can extract file name using \/$[^/?]*?$\? (for one between last slash and question mark) or .*=$.*$$ (for one after last equal sign). If URL is already in variable $URL, use something like that: echo "$URL" |sed -e 's:.*=$.*$$:\1:' (untested, as I'm not on Linux right now).
– Mirek Długosz
Oct 9, 2015 at 11:15
Also, don't forget to click up arrow near answers that you find helpful and accept answer that you... well, accept.
– Mirek Długosz
Oct 9, 2015 at 11:15

Add a comment |

cas · Accepted Answer · 2015-10-08 23:14:30Z

You can use lynx -dump --list-only to extract a list of URLs from the html input.

If you don't already have lynx installed, you'll have to install it using whatever mechanism is normal for your linux or unix system (e.g. apt-get, yum, make install, etc)

e.g.

$ lynx -force_html -dump --listonly shareit1.txt | awk '/^ *[0-9]+\. / { print $2}'
http://gsf-cf.softonic.com/c98/1a8/173dd01ec9001985d81eb5f2023b03280c/LenovoShareIt-win.exe?SD_used=0&channel=WEB&fdh=no&id_file=69703978&instance=softonic_en&type=PROGRAM&Expires=1444364906&Signature=SdKSfTDHY4dG6HVu2--lqt8lRbGK9S1opIDZiSNwvggAAAXB3hESz1G1Y00rU5iLGY5lai0YOJBXhE4y6gvL4uQvCV4U5jzLDU9TmFTxe4xNDrEmkSC95LyGdGSudQKfrWdD06gBlVrqE49AeeotENtdA3SpkmfQGGd1tnjS138_&Key-Pair-Id=APKAJUA62FNWTI37JTGQ&filename=LenovoShareIt-win.exe

Ahmad Awais · Accepted Answer · 2016-08-05 16:58:38Z

0

You can use pup; it's an amazing tool.

Here, see how I download a file and then add particular links from inside of the HTML to a file.

wget http://domain.com -O file.html && pup 'a.className[href] attr{href}' < file.html > links.md

All the links on Domain.com with class name className will be downloaded and saved into a file called links.md.

answered Aug 5, 2016 at 16:58

Ahmad Awais

1012 bronze badges

Add a comment |

Stack Exchange Network

current community

your communities

more stack exchange communities

Trying to grep url from html source in .txt file using sed

5 Answers 5

Your Answer

Not the answer you're looking for? Browse other questions tagged bash sed grep url or ask your own question.

Linked

Hot Network Questions

Trying to grep url from html source in .txt file using sed

5 Answers 5

Your Answer

Sign up or log in

Post as a guest

Not the answer you're looking for? Browse other questions tagged bash sed grep url or ask your own question.

Linked

Related

Hot Network Questions