curl from sitemap.xml?

Question

I would like to curl a list of URLs (Namely as defined in https://example.com/sitemap.xml

The closest I've been able to get to this goal is

xargs curl --referer https://mywebsite.ca --user-agent "It me yo" < listofurls.txt > /dev/null

Which works fine and dandy and hits the URLs I need to but seems unable to work with anything other than a local file. I also would like to make use of a list that exists already and keeps itself maintained (The sitemap.xml).

Answer 1

You'd want to use XSLT to change sitemap.xml into into a list of URLs, one per line. To do this on the command line you can use xsltproc, which is part of libxslt. So you'd do something like:

curl https://example.com/sitemap.xml | xsltproc stylesheet | xargs curl --blah --blah

Creating the XSLT stylesheet is left as an exercise for the reader.