0

is there a way to wget a website and put it's tabular content in .csv?

or maybe a cURL request a webpage, grab it's tabular content represented in numbers that consists of HTML to .csv?

9
  • What do you mean by tabular content and what exact format do you think could represent an HTML page? Jul 7, 2016 at 5:33
  • I want to grab this data from this website: currency.poe.trade/… the content happens to sit on divs
    – unixmiah
    Jul 7, 2016 at 5:36
  • could you use something like ` cat wget www.something.com >> html.csv ` (Notice backticks before wget and after .csv are part of command, not block code)
    – SomeGuy
    Jul 7, 2016 at 5:37
  • 1
    wget 'http://currency.poe.trade/search?league=Prophecy&online=x&want=1&have=4' works but obviously will be in HTML, not CSV, then you can do some data extraction if you want but it can not realistically be done in a shell script. You'd need to use HTML parsing libraries to make something that makes sense such as those available in Python, PHP, Perl, etc. Jul 7, 2016 at 5:49
  • 1
    @MBayley You can escape embedded backticks to prevent it from messing up your code block. cat `wget www.something.com` >> html.csv . But the use of command sub and cat here is pointless anyway.
    – 123
    Jul 7, 2016 at 8:10

1 Answer 1

1

PHP has a class DOMDocument that you can use to retrieve and parse html.

this code will fetch and extract the rows from the webpage. There is still more work necessary to extract the specific items you want but if you are willing to learn some PHP this will get you started

<?php

$html = file_get_contents('http://currency.poe.trade/search?league=Prophecy&online=x&want=1&have=4');

$doc = new DOMDocument;
$doc->loadHTML($html);

$xpath = new DOMXpath($doc);
$rows = $xpath->query('//div[contains(@class, "row")]'); //instance of DOMNodeList
foreach ($rows as $row) {
    // var_dump($row);
    echo "Found {$row->nodeValue}";
}

You can run the code above by copying and pasting in this online PHP interpreter When I run it I get the following sample output (truncated)

Found Currency market // Prophecy ← go to item trades Protip Arrows always point from what you pay to what you get. (You get ⇐ You pay) Currency search Manage your shop Show search form League ProphecyHardcore ProphecyStandardHardcore Online only Off On What do you want? What do you have? Reset .... [more output]

once you've extracted the info you want then its pretty simple to just make each item of interest delimited by a , then insert and newline for each record and then you'll have a CSV file.

Note: for debugging you will need to dump a DOMelement in its HTML/XML markup format. You can use this:

$xml = $domElement->ownerDocument->saveXML($domElement);

or alternatively

$html = $domElement->ownerDocument->saveHTML($domElement);

more background at:
http://php.net/manual/en/class.domelement.php

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.