-1

I'm using ubuntu server (only terminal) and I have a javascript that must be run from a specific website (to avoid cross-domain issues).

Basicly I would follow this steps if I were in a convencional browser:

  1. Open browser
  2. Open a website
  3. Open console and run javascript code

So, how could I do this in terminal?

  • chrome headless + puppeter or casperjs – Gilles Quenot Mar 3 '18 at 18:25
  • Wouldn't Selenium work as well? – Rafael Marinho Mar 3 '18 at 18:33
  • Yes, but it's bloat ware – Gilles Quenot Mar 3 '18 at 18:36
  • There's also firefox's marionette – Gilles Quenot Mar 3 '18 at 18:41
  • Do I need to change my code to work with puppeter? Or there is a way to just open and run it from puppeter? – Rafael Marinho Mar 3 '18 at 20:09
0

A sample realCode™, using the fresh and maintained via and module :

// https://codeburst.io/a-guide-to-automating-scraping-the-web-with-javascript-chrome-puppeteer-node-js-b18efb9e9921

const puppeteer = require('puppeteer');

let scrape = async () => {

    const browser = await puppeteer.launch({headless: true, args: ['--no-sandbox', '--disable-setuid-sandbox']});
    const page = await browser.newPage();

    await page.goto('https://99bitcoins.com/bitcoin-rich-list-top100/#addresses');
    await page.waitFor(3000);

    const result = await page.evaluate(() => {
        var obj = {};
        var data = document.querySelectorAll('table.t99btc-rich-list tr');
        for (var i = 1; i<=100; i++) {
           obj[i] = {
               "hash": data[i].querySelector('td > a').innerText,
               "balance": data[i].querySelector('td:nth-of-type(3)').innerText
           }
        }

        return obj;

    });

    browser.close();
    return result;
};

scrape().then((value) => {
    console.log(JSON.stringify(value, null, 4));
});

Usage :

node top_xbt.js
  • 1
    I accepted it since I followed your suggestion of using puppeteer – Rafael Marinho Mar 4 '18 at 20:03
  • It's the more modern tool and that avoid casperjs specific corner cases – Gilles Quenot Mar 4 '18 at 20:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.