I'm having the following bash script:

URL=`grep -E -m 1 -o "<ExportCatalogResult>(.*)</ExportCatalogResult>" costa_export.xml| sed -e 's,.*<ExportCatalogResult>\([^<]*\)</ExportCatalogResult>.*,\1,g'` &&
echo $URL 
echo -n $url > url.txt &&
wget $(cat url.txt | tr -d " \t\n\r") -O price.zip

The problem is that, when running the script, wget is always downloading an empty file:

http://training.******.net/WAWS_1_9/Catalog/price.zip //echo $URL

Resolving training.******.net (training.*******.net)... 194.**.***.90, 194.**.***.90

Connecting to training.******.net (training.*****.net)|194.**.***.90|:80... connected.

HTTP request sent, awaiting response... 204 Still exporting

Length: 0

Saving to: ‘price.zip’

[ <=>                                                ] 0           --.-K/s   in 0s 

However, the url that is echoed is a valid one (running a wget with that url in the command line will download the zip package).

Why is this happening?

  • 2
    The obvious suspect is that the script URL gets an invisible character included (a newline or a space, for example) that you don't put there when you try it on the command line. – Calle Dybedahl Aug 3 '15 at 9:41
  • @CalleDybedahl and how could I know that. I updated the script to remove all whitespace and newline but without success... – sica07 Aug 3 '15 at 10:08
  • 1
    One way is to print it with delimiters. echo "<$URL>", or so, and see that it looks like you expect. The other is to print it to a file and then check the file with an editor to make sure the content is exactly what you expect. – Calle Dybedahl Aug 3 '15 at 10:24

Two problems jump out at me here:

  1. You haven't quoted your variables, so the shell is potentially mangling the values. I'll give you an alternative suggestion for your script, which draws upon the removal of whitespace.

  2. You're processing XML with Regular Expressions. This will work for now, but be aware that it may well break in the future when something unexpected changes.

Code follows


# ...

# Not even going to attempt to reduce this without seeing your XML
url=$(grep -E -m 1 -o "<ExportCatalogResult>(.*)</ExportCatalogResult>" costa_export.xml| sed -e 's,.*<ExportCatalogResult>\([^<]*\)</ExportCatalogResult>.*,\1,g')

# Debug
echo "raw: $url" >&2

# Strip leading and trailing whitespace
url="${url#"${url%%[![:space:]]*}"}"   # remove leading whitespace
url="${url%"${url##*[![:space:]]}"}"   # remove trailing whitespace

# More debug
echo "now: $url" >&2

# Save the URL
echo -n "$url" > url.txt

# Retrieve the URL contents
wget -O price.zip "$url"

My way of Debugging this situation:

  1. Which shell is executing your script ? Put the "#!" to the shell where it works (you said it works in your command-line)
  2. If you change your script to Directly execute "wget URL", where URL is hardcoded to the output $URL, will it get the file ?
  3. If you take some other URL from some other website , will it work ?
  4. If you have access to the website , can you check the access logs & error logs of the web server ?
  5. What is exit status of wget ? It will help to know what "echo $?" shows, to compare with http://www.gnu.org/software/wget/manual/wget.html#Exit-Status

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.