4

I would like to filter the output of a search on ebay which I've exported to a text file. There are a number of results in the search but I've included just one example at the bottom of this post as I presume I'll be able to use the same method to filter all of the results (keeps things neat on here!). I kind of know the basics of filtering out the url using sed and grep, but I would like the output displayed in a specific way. I want the url followed by a comma, then followed by the price. For example:

http://www.ebay.co.uk/itm/Principles-Of-Modern-Chemistry-International-Edition-Gillis-H-Pat-Oxtoby-Ca-/161952820281?hash=item25b523ec39:g:MEYAAOSwoydWnvT2, £73.69

One thing I will point out at this point is that there are some url's within the text file that are not of use (e.g. http://thumbs.ebaystatic.com/images/g/MEYAAOSwoydWnvT2/s-l225.jpg) but they have a different format to the type i am interested in (i.e. the one I used in the first example). Does anyone know how I can achieve this? Thanks

<h3 class="lvtitle"><a href="http://www.ebay.co.uk/itm/Principles-Of-Modern-Chemistry-International-Edition-Gillis-H-Pat-Oxtoby-Ca-/161952820281?hash=item25b523ec39:g:MEYAAOSwoydWnvT2"  class="vip" title="Click this link to access Principles Of Modern Chemistry, International Edition Gillis, H. Pat; Oxtoby; Ca">Principles Of Modern Chemistry, International Edition Gillis, H. Pat; Oxtoby; Ca</a>^M
                </h3>^M
        <ul class="lvprices left space-zero">^M
^M
        <li class="lvprice prc">^M
                        <span  class="bold bidsold">
                                        £73.69</span>
                                </li>^M
                <li class="lvformat">^M
                        <span >
                                <span class="logoBin" title="Buy it now"></span>
                                        </span>
  • 3
    what you are trying to do is called a screen-scraping and it never is a good idea to spend time on. The page source in HTML, can change without any notice and without changing the effective output, leaving your bash script useless. If you need this data on a continual basis, I suggest investing time and learning about ebay API. Just google the term and you will find pages and pages of results about this API. – MelBurslan Feb 9 '16 at 17:38
  • Thanks, I will check out API but I would still like to know how to do the above as I think I would still find it useful, thanks for the tip though – neilH Feb 9 '16 at 19:29
  • As I am in my lazy mode today, I will not even try to write script here. Let me give you some starting points. 1 determine the start and end lines of one single item listing's details in the page source, using grep -n. 2 using the line numbers from the grep -n command, extract that portion into a new file. 3 using cascaded grep, sed and awk commands, find and extract the URL and the price information. 4 encapsulate the information found with HTML tags and write to a new .html file. – MelBurslan Feb 9 '16 at 20:03
  • Thanks for the response, i'm not sure the the grep -n will work as the number of lines between the url's and prices differ within the file. Sometimes it's 5 lines sometimes, it's 6 or 7. – neilH Feb 9 '16 at 23:46
  • You need to know patterns for the beginning line and end line for weach item. run grep -n against those and use the two numbers as the range to your sed command. Otherwise, as I said, html is very unpredictable – MelBurslan Feb 10 '16 at 0:11
3

The best way to get at data from eBay is through their API. This being said, sometimes all you have is HTML, so I'll cover that in my answer.

Don't even try to extract information from HTML with tools like sed and grep. It's hard to do when it works at all, and extremely brittle. This way lies madness.

If you have to parse HTML, use a tool for parsing HTML, such as Python's BeautifulSoup library, Perl's HTML::TreeBuilder, Ruby's nokogiri, etc.

#!/usr/bin/env python2
import codecs, sys, BeautifulSoup
html = BeautifulSoup.BeautifulSoup(codecs.open(sys.argv[1], "r", "utf-8").read())
for lv in html.findAll("h3", "lvtitle"):
    url = lv.find("a")["href"]
    bid = lv.findNextSibling("ul").find("span", "bidsold").text.strip()
    print(url, bid)
  • ok, thanks, i'm going to give this a go and i'll get back to you. I've not heard of any of those parsing tools before. In the code you have printed in your response where do I state the .txt file with my html code? cheers – neilH Feb 12 '16 at 20:57
  • @bms9nmh I posted a script. You pass the input file as an argument to this script. – Gilles 'SO- stop being evil' Feb 12 '16 at 20:58
  • Good answer, directed me into learning about API's and ready made parsers which will prove useful. – neilH Feb 22 '16 at 17:22
0

Here's a quick and dirty solution:

sed -n 's/^.*href="\([^"]*\)".*$/\1/p; s/^.*\(£[0-9.]*\).*$/\1/p' |
awk 'NR % 2{ printf "%s, ", $0; next} {print}' 

It simply extracts what looks like hrefs and what looks like prices, hoping you'll get an alternating sequence of urls and prices. Then joins consecutive lines to get the format you want.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.