Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

We have a vendor that sends us xml feeds of new items with the appropriate path and image name. I created script that will strip the web address path for the image, but if I try to obtain these with wget I get a protocol error. I tried using wget -i list.txt. If I use other tools similar I do not have an issue. What would be causing the issue and what is best suggested?

share|improve this question
Do you use single quotes around the address in wget invocation? – Deer Hunter Jan 4 '13 at 18:24
no in the tutorial and manual for wget it did not state to use single quotes. – Darth_Vader Jan 4 '13 at 18:30
are you sure that list contains proper URL's ? – Rahul Patil Jan 4 '13 at 18:30
positive, I have randomly tested URLs within Firefox and Chrome. So far I have randomly tested 52 image URLS out of a 2000 list document. I have no issues with the stripper to obtain the address only – Darth_Vader Jan 4 '13 at 18:33
up vote 1 down vote accepted

i think this will work

grep -oP "http:\//[a-zA-Z0-9\-\.]+\.[a-zA-Z]{2,3}(:[a-zA-Z0-9]*)?/?([a-zA-Z0-9\-\._\?\,\'/\\\+&%\$#\=~])*.jpg" inputfile  | xargs -n1 wget

or if you have list then you can use

cat list.txt | xargs -n1 wget 
share|improve this answer
After using cat I get a 400 Bad request – Darth_Vader Jan 4 '13 at 18:32
@graphicsman means your list.txt file does not contain proper URLs, you can use grep with url regex as i mention to your xml file – Rahul Patil Jan 4 '13 at 18:33
no. The server appears to have gone down. Calling technical support. Now I cannot get the images that once displayed in the browser. – Darth_Vader Jan 4 '13 at 18:36
@graphicsman if you post that error with that url, then it would be helpful to understand your issue.. – Rahul Patil Jan 4 '13 at 18:36
@graphicsman if you post that error with screen-shot , it will help to resolve issue. – Rahul Patil Jan 5 '13 at 5:33

Without seeing a sample of the input and at least the core of your script, I can only speculate that the URLs are being interfered with by malevolent space yetis. They are notorious for this irresponsible behaviour - your only hope is to call Area 51 tech support.

Or maybe provide sample and script here. A crazy idea, but it might help. Anonymise or obfuscate them if necessary, but without messing up the structure of the XML file.

I'm going to guess that you're using a simple regexp and point out that as a general rule, "parsing" XML with regexps is not a good idea - it's fragile and easily confused by valid but unexpected input...use an xml parsing library or function or even a tool like xml2 to extract the URLs from the XML files.

Note also that, as hinted by Deer Hunter above, if the URLs contain any spaces or punctuation characters with special meaning to the shell (e.g &, ?, *, and many others) you will need to quote or escape the URL before using it on a wget command line. As a very simple example, use wget "$URL" rather than just wget $URL - note the double-quotes around the $URL variable.

share|improve this answer

after much review we found the network was an issue. The wget script would work for other sites and the links downloaded with window based tools such as jdownloader. For some reason the Linux based side would not access the NT web server from the vendor. We have installed winbind but the an examination of out network is in order.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.