0

I don't understand...

The actual download link appears to not be a http link but rather some Javascript action?

javascript:SendFileDownloadCall('PRODIMAGES.CIF.zip','PRODIMAGES.CIF.zip');

So after downloading manually, I went to the browser's download history to copy the direct link https://au.ingrammicro.com/_layouts/CommerceServer/IM/FileDownload.aspx?DisplayName=STD_FULL_FILEFEED.TXT&FileName=STDPRICE_FULL.TXT.zip

And I fed the URL into wget, along with my website credentials:

wget -q --user=XXXX --password=XXXX "https://au.ingrammicro.com/_layouts/CommerceServer/IM/FileDownload.aspx?DisplayName=STD_FULL_FILEFEED.TXT&FileName=STDPRICE_FULL.TXT.zip" -o STDPRICE.zip

Later, I see that adding the --user and --password make no difference so I omit:

[root@server datafiles]# wget "https://au.ingrammicro.com/_layouts/CommerceServer/IM/FileDownload.aspx?DisplayName=STD_FULL_FILEFEED.TXT&FileName=STDPRICE_FULL" -O STDPRICE.zip
--2019-09-15 19:53:29--  https://au.ingrammicro.com/_layouts/CommerceServer/IM/FileDownload.aspx?DisplayName=STD_FULL_FILEFEED.TXT&FileName=STDPRICE_FULL
Resolving au.ingrammicro.com (au.ingrammicro.com)... 104.98.45.15
Connecting to au.ingrammicro.com (au.ingrammicro.com)|104.98.45.15|:443... connected.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location: /_layouts/CommerceServer/IM/Login.aspx?ReturnUrl=%2f_layouts%2fCommerceServer%2fIM%2fFileDownload.aspx%3fDisplayName%3dSTD_FULL_FILEFEED.TXT%26FileName%3dSTDPRICE_FULL [following]
--2019-09-15 19:53:29--  https://au.ingrammicro.com/_layouts/CommerceServer/IM/Login.aspx?ReturnUrl=%2f_layouts%2fCommerceServer%2fIM%2fFileDownload.aspx%3fDisplayName%3dSTD_FULL_FILEFEED.TXT%26FileName%3dSTDPRICE_FULL
Reusing existing connection to au.ingrammicro.com:443.
HTTP request sent, awaiting response... 200 OK
Length: 85341 (83K) [text/html]
Saving to: ‘STDPRICE.zip’

100%[===================================================================================================================================================================================================>] 85,341       405KB/s   in 0.2s

2019-09-15 19:53:30 (405 KB/s) - ‘STDPRICE.zip’ saved [85341/85341]

Anyway, instead of getting the equivalent file to the one that I get by human clicking and downloading from the website I get an implausibly smaller file.

Confirming my fears, when I try to unzip, I get:

$ [root@server datafiles]# unzip STDPRICE.zip
Archive:  STDPRICE.zip
  End-of-central-directory signature not found.  Either this file is not
  a zipfile, or it constitutes one disk of a multi-part archive.  In the
  latter case the central directory and zipfile comment will be found on
  the last disk(s) of this archive.
unzip:  cannot find zipfile directory in one of STDPRICE.zip or
        STDPRICE.zip.zip, and cannot find STDPRICE.zip.ZIP, period.

On file inspection:

$ [root@server datafiles]# file STDPRICE.zip
STDPRICE.zip: HTML document, UTF-8 Unicode text, with very long lines, with CRLF line terminators

So wget has actually downloaded a HTML file which is being presented as a .txt.zip file? wth Can somebody enlighten me?

1

The site is redirecting you to the login page:

HTTP request sent, awaiting response... 302 Moved Temporarily
Location: /_layouts/CommerceServer/IM/Login.aspx?ReturnUrl=%2f_layouts%2fCommerceServer%2fIM%2fFileDownload.aspx%3fDisplayName%3dSTD_FULL_FILEFEED.TXT%26FileName%3dSTDPRICE_FULL [following]
--2019-09-15 19:53:29--  https://au.ingrammicro.com/_layouts/CommerceServer/IM/Login.aspx?ReturnUrl=%2f_layouts%2fCommerceServer%2fIM%2fFileDownload.aspx%3fDisplayName%3dSTD_FULL_FILEFEED.TXT%26FileName%3dSTDPRICE_FULL

It probably doesn't accept the credentials you provided as basic auth (which is what wget is sending) but instead using session cookies. You can try extracting the cookies from your browser (while logged in) and sending them with wget (--load-cookies). They might also be keeping an eye on other aspects of the request (user agent for example) that you might try modifying also.

If you can use curl instead, open the inspector (Ctrl+Shift+I), go to the network tab, download the file, right-click the entry for the download in the request list, mouse over "Copy", then select "Copy as cURL", and the command now on your clipboard will include the cookies.

| improve this answer | |
  • 1
    The second step there is what you want, yes, but the first step would be extracting the cookies from your browser (since we don't know how the site accepts credentials). If you can use curl instead, open the inspector (Ctrl+Shift+I), go to the network tab, download the file, right-click the entry in the network tab, select "Copy", then "Copy as cURL, and the command line copied will include the cookies. – gary Sep 15 '19 at 18:18
  • 1
    The command is fully formed when you copy from the network tab in the inspector. Paste it into your terminal and press enter. – gary Sep 15 '19 at 18:20
  • 1
    You'd probably have to write it yourself. curl tends to be more popular in general than wget. You could get your cookies into your wget command by translating the -H 'Cookie: ...' part of the curl command to --header 'Cookie: ...' for wget. – gary Sep 15 '19 at 18:52
  • 1
    Thanks! Was able to get it working as per your suggestion: wget -v --content-disposition --header 'Cookie: ...' https://au.ingrammicro.com/_layouts/CommerceServer/IM/Login.aspx?ReturnUrl=%2f_layouts%2fCommerceServer%2fIM%2fFileDownload.aspx%3fDisplayName%3dSTD_FULL_FILEFEED.TXT%26FileName%3dSTDPRICE_FULL.TXT.zip – ptrcao Sep 15 '19 at 19:28
  • 1
    Yes they will definitely expire. They'll only be useful for probably a few hours at a time. If you need them long-term you'll either need to work out how to submit your credentials to their login form and capture the cookies for a subsequent request (like the first step from the link in your first comment), or switch to something more sophisticated like a headless browser. – gary Sep 15 '19 at 23:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.