Wget images issue from api_key

Question

We have a vendor that sends us xml feeds of new items with the appropriate path and image name. I created script that will strip the web address path for the image, but if I try to obtain these with wget I get a protocol error. I tried using wget -i list.txt. If I use other tools similar I do not have an issue. What would be causing the issue and what is best suggested?

Answer 1

i think this will work

grep -oP "http:\//[a-zA-Z0-9\-\.]+\.[a-zA-Z]{2,3}(:[a-zA-Z0-9]*)?/?([a-zA-Z0-9\-\._\?\,\'/\\\+&%\$#\=~])*.jpg" inputfile  | xargs -n1 wget

or if you have list then you can use

cat list.txt | xargs -n1 wget 

Answer 2

Without seeing a sample of the input and at least the core of your script, I can only speculate that the URLs are being interfered with by malevolent space yetis. They are notorious for this irresponsible behaviour - your only hope is to call Area 51 tech support.

Or maybe provide sample and script here. A crazy idea, but it might help. Anonymise or obfuscate them if necessary, but without messing up the structure of the XML file.

I'm going to guess that you're using a simple regexp and point out that as a general rule, "parsing" XML with regexps is not a good idea - it's fragile and easily confused by valid but unexpected input...use an xml parsing library or function or even a tool like xml2 to extract the URLs from the XML files.

Note also that, as hinted by Deer Hunter above, if the URLs contain any spaces or punctuation characters with special meaning to the shell (e.g &, ?, *, and many others) you will need to quote or escape the URL before using it on a wget command line. As a very simple example, use wget "$URL" rather than just wget $URL - note the double-quotes around the $URL variable.

Answer 3

after much review we found the network was an issue. The wget script would work for other sites and the links downloaded with window based tools such as jdownloader. For some reason the Linux based side would not access the NT web server from the vendor. We have installed winbind but the an examination of out network is in order.