-1

I've been crawling over various articles for the last 8 hours and haven't made much headway. I've tried httrack, wget, lynx & cURL.

A website I am a premium member of allows download of content at fast speeds once you pay and slow speeds for free members. All links are generated by the server and are one time download links to stop link sharing.

After about 3 hours I managed to figure out how to get wget to save my cookies file. Now my issue is when I try to download the files. The following wget command downloads all of the product pages but not the actual files. There is an <a> tag on each individual page linking to the downloadable file but wget isn't grabbing these.

wget --mirror --adjust-extension --page-requisites --execute=robots=off --wait=30 --random-wait --convert-links --user-agent=Mozilla --no-parent --recursive --load-cookies=my-cookies.txt --no-clobber https://www.mysite.net/directory/product

I have a feeling it's something simple I am missing. After all, I have been working trying to get this to work for over 8 hours. :/

EDIT: I tried turning off Javascript per Anthon's suggestion but Javascript doesn't seem to play any roll in downloading files. Further examination of the website code shows a simple url tag is used, <a>.

  • 4
    Instead of thanks and advance notices of your appreciation, you should always include an interrogative sentence in your post (In English easily identifiable by the closing question mark (?)). Have you tried to work with the site in your browser with JavaScript completely turned off? If you did can you confirm (by updating your post, not by answering in a comment, that that works? – Anthon Aug 22 '16 at 6:50
  • Please show an example of an <a> tag that's not being followed. Is the target definitely on the same site, and under the same path? (You may need to remove the --convert-links option while debugging this.) – JigglyNaga Aug 23 '16 at 7:48
  • Example: https://www.somewebsite.net/download/member/384486/eyJpdiI6ImhqMlpUSnplXC9hMmt3S1gwVEVBUmJ3PT0iLCJ2YWx1ZSI6ImxjTUNyQjRWTWZIXC94RjJuSEtpSHpBPT0iLCJtYWMiOiJlM2I1MWZlYmZlZjdmMjBjMTU5NGViN2JiZjNmOGFhMmJjMzcwYTI3ZTRkNzJmNGE5MjMwZDAzNTVhZDY2Y2IwIn0= – Jared Kindred Aug 23 '16 at 7:52
  • It seems the files reside on 'us3.somewebsite.net' – Jared Kindred Aug 23 '16 at 8:07
2

I use cliget (a firefox addon) that can create the command lines for wget & cURL which can then be used on the command line. See if this works:

https://addons.mozilla.org/en-US/firefox/addon/cliget/

One reason this may not be working (as @Anthon points out) is that the download link is actually executing some javascript that computes the actual file location - trapping that call is the solution. For automated download of that sort, one can use selenium + python - a bit convoluted approach though & possibly not suitable for this site.

  • This works beautifully, however, I have roughly 4 - 500 small files to download from the site and each link involves a randomly generated string. Whenever I try to use wget to pull the download links it never seems to work. – Jared Kindred Aug 23 '16 at 7:49
  • @JaredKindred: In that case you might have to look at the (possibly) JavaScript used to create the "random" string. And then execute the JS using a web-robot like Selenium (with a Python controller). Essentially, you set the parameters that the JS expects, submit the form, wait for the "download" button to show up & click on it. – SACHIN GARG Aug 23 '16 at 19:14
  • I used a plugin in chrome to completely shut JavaScript off in the browser. The website loads completely fine and the download buttons work without JavaScript being enabled. Correct me if I'm wrong but doesn't that mean they aren't generated on the user end? – Jared Kindred Aug 24 '16 at 3:53
  • Yes, in that case it possibly (because I am not an expert) does mean they are not generated at the user end, and the server is generating them using some logic. – SACHIN GARG Aug 24 '16 at 3:58
  • Okay after adding options -I & -D I was able to get all the page links for the url downloads. However, the website is still showing me as a guest user instead of logged in. I checked out cliget and found that it works really well, for single file downloads. I copied the cookie info that cliget was using but it still shows me as a guest user for the whole site. – Jared Kindred Aug 24 '16 at 5:27
0

You might try adding wget option -H or the long form

--span-hosts

so your --recursive allows wget to get the files from the second site given in the <a> href. You might also try

--referer=originurl

where originurl is the full url of the page with all the anchors.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.