0

I would like to curl a list of URLs (Namely as defined in https://example.com/sitemap.xml

The closest I've been able to get to this goal is

xargs curl --referer https://mywebsite.ca --user-agent "It me yo" < listofurls.txt > /dev/null

Which works fine and dandy and hits the URLs I need to but seems unable to work with anything other than a local file. I also would like to make use of a list that exists already and keeps itself maintained (The sitemap.xml).

0

You'd want to use XSLT to change sitemap.xml into into a list of URLs, one per line. To do this on the command line you can use xsltproc, which is part of libxslt. So you'd do something like:

curl https://example.com/sitemap.xml | xsltproc stylesheet | xargs curl --blah --blah

Creating the XSLT stylesheet is left as an exercise for the reader.

  • This definitely looks like what I'm after. Thank you. It looks like I already have xsltproc installed on my machine. I'll have to experiment/learn/play with the stylesheet aspect. It seems Yoast has released stylesheets meant to work with their sitemaps already. – Jay Aug 27 '18 at 16:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.