0

I have list of wget command as shown below:

wget -q -nH --no-check-certificate --cut-dirs=5  -r -l0 -c -N -np -R 'index*'  -erobots=off --retr-symlinks https://heasarc.gsfc.nasa.gov/FTP/swift/data/obs/2009_12//00030352160/auxil/
wget -q -nH --no-check-certificate --cut-dirs=5  -r -l0 -c -N -np -R 'index*'  -erobots=off --retr-symlinks https://heasarc.gsfc.nasa.gov/FTP/swift/data/obs/2020_11//00031630169/xrt/
wget -q -nH --no-check-certificate --cut-dirs=5  -r -l0 -c -N -np -R 'index*'  -erobots=off --retr-symlinks https://heasarc.gsfc.nasa.gov/FTP/swift/data/obs/2017_03//00034228170/auxil/
wget -q -nH --no-check-certificate --cut-dirs=5  -r -l0 -c -N -np -R 'index*'  -erobots=off --retr-symlinks https://heasarc.gsfc.nasa.gov/FTP/swift/data/obs/2020_03//00031630125/log/
wget -q -nH --no-check-certificate --cut-dirs=5  -r -l0 -c -N -np -R 'index*'  -erobots=off --retr-symlinks https://heasarc.gsfc.nasa.gov/FTP/swift/data/obs/2008_11//00030352101/auxil/
wget -q -nH --no-check-certificate --cut-dirs=5  -r -l0 -c -N -np -R 'index*'  -erobots=off --retr-symlinks https://heasarc.gsfc.nasa.gov/FTP/swift/data/obs/2014_04//00035014140/log/
wget -q -nH --no-check-certificate --cut-dirs=5  -r -l0 -c -N -np -R 'index*'  -erobots=off --retr-symlinks https://heasarc.gsfc.nasa.gov/FTP/swift/data/obs/2017_03//00034228172/auxil/
wget -q -nH --no-check-certificate --cut-dirs=5  -r -l0 -c -N -np -R 'index*'  -erobots=off --retr-symlinks https://heasarc.gsfc.nasa.gov/FTP/swift/data/obs/2017_04//00034228177/auxil/
wget -q -nH --no-check-certificate --cut-dirs=5  -r -l0 -c -N -np -R 'index*'  -erobots=off --retr-symlinks https://heasarc.gsfc.nasa.gov/FTP/swift/data/obs/2014_04//00035014133/auxil/
wget -q -nH --no-check-certificate --cut-dirs=5  -r -l0 -c -N -np -R 'index*'  -erobots=off --retr-symlinks https://heasarc.gsfc.nasa.gov/FTP/swift/data/obs/2019_12//00031630093/auxil/

These are saved in a filename.txt. How to download? How can I see the progress of download also in log file?

1
  • To answer the first question, just run sh filename.txt. Jun 25, 2021 at 5:39

1 Answer 1

0

There is a very simple solution and a simple solution.

As berndbausch said, you could either just run bash filename.txt as shellscripts are basically texfiles with lists of commands, just as your textfile.

Or you could do something more clever, but maybe not that useful for this usecase: If you got a list of links (https://heasarc.gsfc.nasa.gov/FTP/swift/data/obs/2009_12//00030352160/auxil/, https://heasarc.gsfc.nasa.gov/FTP/swift/data/obs/2020_11//00031630169/xrt/, ...) you can wirte them in you textfile, and the use that textfile as an input for wget.

textfile.txt:

https://heasarc.gsfc.nasa.gov/FTP/swift/data/obs/2009_12//00030352160/auxil/
https://heasarc.gsfc.nasa.gov/FTP/swift/data/obs/2020_11//00031630169/xrt/
https://heasarc.gsfc.nasa.gov/FTP/swift/data/obs/2017_03//00034228170/auxil/
https://heasarc.gsfc.nasa.gov/FTP/swift/data/obs/2020_03//00031630125/log/
https://heasarc.gsfc.nasa.gov/FTP/swift/data/obs/2008_11//00030352101/auxil/
https://heasarc.gsfc.nasa.gov/FTP/swift/data/obs/2014_04//00035014140/log/
https://heasarc.gsfc.nasa.gov/FTP/swift/data/obs/2017_03//00034228172/auxil/
https://heasarc.gsfc.nasa.gov/FTP/swift/data/obs/2017_04//00034228177/auxil/
https://heasarc.gsfc.nasa.gov/FTP/swift/data/obs/2014_04//00035014133/auxil/
https://heasarc.gsfc.nasa.gov/FTP/swift/data/obs/2019_12//00031630093/auxil

an then: wget -q -nH --no-check-certificate --cut-dirs=5 -r -l0 -c -N -np -R 'index*' -erobots=off --retr-symlinks -i textfile.txt

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.