2

I am trying to download some files from a services. The files are found in an XML file. The XML file can have a single file or several files to download. However, now I have a problem with my script. I do not know how to split string from XMLLINT into array so that I can download each file individually.

I need to split the string into several variables and then download each file of the URL string.

However the file 201701_1 do not repeat and hence, I download them using curl without any problems. But the files coverage.zip repeat and they become overwritten by curl. I do: Then I do curl to download individual files.

curl -O -b cookie $URL 

At the moment, my script is as follows:

while read edition; do   XML="<?xml version=\"1.0\"
encoding=\"UTF-8\"?> <download-area>   <files>
    <file>
      <url>https://google.com/411/201701_01_01.zip</url>
    </file>
    <file>
      <url>https://google.com/411/201701_01_02.zip</url>
    </file>   </files> </download-area>
    "
    URL=$(echo $XML | xmllint --xpath \
    "/*[name()='download-area']/*[name()='files']/*[name()='file']/*[name()='url']/text()" -)

    echo "URL:: " $URL

done < $LATEST_EDITION

LATEST_EDITION is a simply a file with lines.

My question is:: How can I split VAR_1 and VAR_2 into several URLs so that I can download them individually? How can I prevent coverage.zip from being overwritten?

  • You can use https:// as a delimiter for splitting, but iit would be smarter not to join them in the first place. Have you tried to make xmllint produce output with delimiters between the URLs? – Philippos Nov 20 '17 at 8:04
  • your current script does not use any VAR_1 and VAR_2. Update your question – RomanPerekhrest Nov 20 '17 at 8:14
  • I have removed VAR_1 and VAR_2 which brought confusion to the question. – Noel Alex Makumuli Nov 20 '17 at 11:19
  • You might want to include an example of the XML data that you're trying to parse and what the expected output/result would be for that example. – igal Nov 21 '17 at 3:01
1

Try something like:

declate -a url_array
url_array=(`echo $XML | grep -o "http.*zip" | tr '\n' ' '`)
  • Why translate the \n to ' '? Both space and newline are in the default value of $IFS. It would make more sense to remove space from $IFS. Also, when using the split+glob operator, it's good practice to disable the glob part when you don't need it. It makes little sense to invoke split+glob upon the expansion of $XML, so that $XML should be quoted. – Stéphane Chazelas Nov 20 '17 at 11:24
1

xmllint isn't a great tool for slicing and dicing XML. To solve your two problems (parsing XML, and ensuring unique URLs, I think?) in a robust way using bash and xmlstarlet:

#!/bin/bash
XML='<?xml version="1.0" encoding="UTF-8"?>
<download-area>
  <files>
    <file>
      <url>https://google.com/411/201701_01_01.zip</url>
    </file>
    <file>
      <url>https://google.com/411/201701_01_02.zip</url>
    </file>
  </files>
</download-area>'

# IFS=$'\n'   ## required if URLs contains spaces
urls=( $(xml select -t -m  "/download-area/files/file" -v url -nl  <<< $XML ) )

declare -A unique  # associative array
for uu in ${urls[*]}; do let unique[$uu]++; done

for uu in "${!unique[@]}"; do
  printf "URL is %s\n" ${uu}
done

This uses xmlstarlet in select mode, with a template (-t) which matches (-m) an xpath from which it selects the value (-v) of the url node and adds a newline (-nl) after each one. (xmlstarlet is even more flexible than this, you can use -v multiple times and -o to add arbitrary text wherever required.)

This also uses the <<< redirection which saves an echo/pipe.

The URLs are stored in a normal indexed array urls. Next, loop over the array to store the URLs as the keys in an associative array — this solves the uniqueness problem (and the count of occurrences is kept as the value for each entry).

If you're not familiar with bash's associative arrays the second loop needs some additional explanation. The expression "${unique[@]}" expands all the values of an array, this uses "${!unique[@]}" which instead expands all the indexes of an array, this should make sense if you dump the array data with declare -p unique:

declare -A unique=([https://google.com/411/201701_01_01.zip]="1" 
                   [https://google.com/411/201701_01_02.zip]="1" )

You can even do this all in one loop, though it's a little harder to understand perhaps:

while read line; do
  [[ -n "$line" ]] && let unique[$line]++ 
done < <(xml sel -t -m  "/download-area/files/file" -v url  -nl <<< $XML)

XMLstarlet may be installed as either xml or xmlstarlet

  • Thanks for this.. I just managed to put an almost similar implementation.. I also learned that XMLLINT is not the best tool for slicing and dicing with XML.. Kudos..!! – Noel Alex Makumuli Nov 20 '17 at 11:24
1

xmllint is pretty useless to extract information from XML documents. You may want to consider xmlstarlet or xml_grep (from perl's XML::Twig) or xml2.

With xmllint, you could still extract one string at a time with:

VAR1=$(printf '%s\n' "$XML" |
  xmllint --xpath '/download-area/files/file[1]/url/text()' -)
VAR2=$(printf '%s\n' "$XML" |
  xmllint --xpath '/download-area/files/file[2]/url/text()' -)

For values like here not containing newline characters, you can use bash's readarray as:

readarray -t var < <(
  xmlstarlet sel -t -v /download-area/files/file/url  <<< "$XML")

Or

readarray -t var < <(
  xml2 <<< "$XML" | sed -n 's|^/download-area/files/file/url=||p')

Or:

readarray -t var < <(
  xml_grep --text_only /download-area/files/file/url <<< "$URL")
  • Huh? xmllint is an excellent utility for extracting information from a structured document using the XPath expression language. Where it "fails" is that it does not produce the output in a format that people can easily use with procedural languages such as Bash – fpmurphy Nov 21 '17 at 2:05
0

Consider using sedto parse the output from xmllint. Note the shortened XPath expression!

URL=$( echo $XML | xmllint --xpath "//url" - | sed -e 's/<url>//g' -e 's/<\/url>/\n/g' )

printf "%s\n" "$URL"

This will output the URLs one per line

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.