0

This question already has an answer here:

I have a bash script that scrapes a list of urls for links to various kinds of documents. At the end, the script uses wget to download the files. However, I'm having trouble with filenames containing white space in the name: wget ends the url at the space. Is there some way to use sed or something to change the white space to %20 here? Or some other solution? This is my code:

for url in $(cat download.md)
do
    lynx --listonly --dump $url | \
awk '/\.(pdf|doc|docx|odt)$/{ sub("^[ ]*[0-9]+.[ ]*","",$0); print}'

done > ~/links.txt

for i in $( cat ~/links.txt ); do wget $i; 
done

marked as duplicate by Jesse_b, Isaac, msp9011, countermode, schily Aug 30 '18 at 12:26

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 1
    Have you tried quoting your variables, e.g., wget "$i"? – dsstorefile1 Aug 29 '18 at 18:50
  • I did, but it didn't work. – Linter Aug 29 '18 at 18:52
  • wget "${i// /%20}"? – DopeGhoti Aug 29 '18 at 18:52
  • @DopeGhoti also fails to download because the url is cut off at the space. – Linter Aug 29 '18 at 18:58
  • Based on the for i in $( cat ... ) habit, I'd also recommend reading: unix.stackexchange.com/questions/169716/… – Jeff Schaller Aug 29 '18 at 19:06
1

You lost the full filenames during the $( ... ) command substitution. Just ask wget to read the file directly:

wget -i ~/links.txt
  • Almost. This downloaded 4 copies of everything... – Linter Aug 29 '18 at 19:07
  • Add --debug to see what exactly it's doing; perhaps you have 4 copies in the file? – Jeff Schaller Aug 29 '18 at 19:07
  • I got it! Thank you, Jeff. You helped me so much over the past two days! – Linter Aug 29 '18 at 19:14

Not the answer you're looking for? Browse other questions tagged or ask your own question.