0

I am trying to extract the artists of this page. I have tried many variations of

wget https://northside.dk/artister/

and

wget --random-wait -r -p -e robots=off -U mozilla https://northside.dk/artister/

but I only get

<head>
  <meta charset="UTF-8">

  <meta name="google-site-verification" content="clAYDF67yhmgMMhQ8tcJTXpuo4TGpmHSbo4RyIMu6vY" />

  <meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=0"/>
  <meta name="apple-mobile-web-app-capable" content="yes">
  <meta name="apple-mobile-web-app-status-bar-style" content="black-translucent">
  <link rel="shortcut icon" href="/img/favicon_43f15e.png">

  <title>NorthSide - 4. - 6. juni 2020</title>

  <script id="CookieConsent" src="https://policy.app.cookieinformation.com/uc.js" type="text/javascript"></script>

  <!-- Google Tag Manager -->
  <script>(function (w, d, s, l, i) {
      w[l] = w[l] || []; w[l].push({
        'gtm.start':
          new Date().getTime(), event: 'gtm.js'
      }); var f = d.getElementsByTagName(s)[0],
        j = d.createElement(s), dl = l != 'dataLayer' ? '&l=' + l : ''; j.async = true; j.src =
          'https://www.googletagmanager.com/gtm.js?id=' + i + dl; f.parentNode.insertBefore(j, f);
    })(window, document, 'script', 'dataLayer', 'GTM-K4RZFD');

    if ( typeof "ga" === "function" ) {
      ga('require', 'linker');
    }
    
    if ( typeof "gtag" === "function" ) {
      gtag('config', 'UA-22269830-1', {
        'linker': {
          'domains': ['northside.dk', 'ticketmaster.dk', 'tmmikrobetaling.dk']
        }
      });
    }
  </script>
  <!-- End Google Tag Manager -->
  
  <!-- Google Tag Manager (noscript) -->
  <noscript><iframe src="https://www.googletagmanager.com/ns.html?id=GTM-K4RZFD" height="0" width="0" style="display:none;visibility:hidden"></iframe></noscript>
  <!-- End Google Tag Manager (noscript) -->
<link rel="shortcut icon" href="/favicon.png"><script type="text/javascript" src="/main_611e89.js"></script></head>

<body>
  <div id="app-mount">
    <div class="marble-loader" id="loader"></div>
    <h1 id="loading-status">Booking music</h1>
  </div>
</body>

instead of the expected output, I got from the firefox inspector with blocks like:

<a class="archive-grid-item-shell” href="/artister/3447" data-reactid=".0.1.4.1.0.0.0.$3447.1">
  <div class="archive-grid-iten-content" data-reactid=".0.1.4.1.6.0.0.53447.1.0">
    <div class="grid-iten-label” data-reactid=".0.1.4.1.0.8.0.$3447.1.0.0"> w/</div>
    <div class="grid-iten-header’ data-reactid=".0.1.4.1.0.0.0.53447.1.0.1">
      <span class="text-with-background" data-reactid=".0.1.4.1.0.0.0.53447.1.0.1.0">Clara</span>
    </div>
  </div>
</a>

where all objects are captured. I even tried with the headerless browser, lynx, but with the same result as wget.

What am i doing wrong or is the page being run in such a way that I can't get the content with wget?

3
  • 2
    wget can only get elements that are included directly on the page. Obviously, this page loads a javascript, which loads additional content and puts it into the page dynamically. wget cannot run javascript code, and therefore cannot determine, which elements are loaded from javascript after a page is loaded.
    – nobody
    Jan 4, 2020 at 15:01
  • You can use something like selenium (e.g., via python) to programatically interact with real browsers. This lets you access javascript-generated content.
    – larsks
    Jan 4, 2020 at 15:33
  • Yeah was thinking Js was the culprit, but thanks for your answer! I'll see if selenium can get the job done Jan 4, 2020 at 16:04

2 Answers 2

1

try this way:

wget -q -O - "https://api.northside.dk/wp-json/wp/v2/cpt-artist/?orderby=menu_order&order=asc&per_page=100" | grep -oP '"raw":.*?[^\\]"'
"raw":"Clara"
"raw":"Folkeklubben"
"raw":"Franc Moody"
"raw":"Green Day"
"raw":"Hans Philip"
"raw":"Johnny Marr"
"raw":"Jung"
"raw":"Kashmir"
"raw":"Lukas Graham"
"raw":"Mags"
"raw":"Mekdes"
"raw":"Mew"
"raw":"Robyn"
"raw":"Spleen United"
"raw":"Weezer"
"raw":"White Lies"

update #1

in browser you start developer tools, in chrome you can do it via F12 then choose the network bookmark and enter the website address

at this point you should see all requests that the browser sends to the server, and their responses

when you follow the network traffic well, you will see a request responsible for downloading the necessary data

you can choose each request to check its query parameters, response, status, etc.

enter image description here

2
  • The interesting parth here is where you got the url from? Jan 7, 2020 at 4:32
  • By elaborating I will be able to mark this as the best solution available as it indeed does solve it. But I really need to understand how you got there. Jan 7, 2020 at 9:44
0

as @nobody said:

wget can only get elements that are included directly on the page. Obviously, this page loads a javascript, which loads additional content and puts it into the page dynamically. wget cannot run javascript code, and therefore cannot determine, which elements are loaded from javascript after a page is loaded. – nobody yesterday

The material could be extracted successfully with selenium in python.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .