2

Lets say there's a ftp site with a bunch of files I want to download - ftp://ftp.example.com, and they allow for an "anonymous login" over FTP.

Suppose that ftp://ftp.example.com has

file1.zip, 
file2.zip, 
... ,
file100.zip

Also suppose that I have a list of the filepaths I want to download all in a bash variable like so:

paths="ftp://ftp.example.com/file1.zip ftp://ftp.example.com/file7.zip"

Is there a way to do wget $paths (i.e. download my list of two files) without having to see wget do for every file:

Connecting to ftp.example.com|10.10.10.10|:21... connected.
Logging in as anonymous ... Logged In!

where wget logs in to the authenticates with the site once for every single file I download?

  • As far as I can tell from the source, nope. Wget will always close the connection at the end of each file for FTP. Honestly, I have no idea why. If you drop a mail on bug-wget@gnu.org, we'll try and help you out / fix the code in Wget for this case – darnir Oct 20 '18 at 9:42
1

Alternatively, you could use plain old ftp. Set up a stanza for ftp.example.com in your ~/.netrc file:

machine ftp.example.com
  login ftp
  password anonymous

... ensure that the file is not readable by anyone else:

chmod 600 ~/.netrc

Then send ftp commands via a here-document to retrieve the files:

ftp ftp.example.com <<'EOF'
prompt
mget file*.zip
bye
EOF

Some ftp clients may support the -i option to disable prompting for mget, negating the need for the prompt command in the list.

Using this will establish a single ftp session for all of the file transfers.

0

I think you want tip number 9 on this page.

Basically, download multiple files with wget -i,

$ cat > download-file-list.txt
URL1
URL2
URL3
URL4

$ wget-i download-file-list.txt
  • Doesn't quite work. Wget still displays the ... connected ... Logged In! messages when I write file paths out to temp.txt then do wget -i temp.txt. I timed the difference for ~20 files transferred - results were 3 mins 3 seconds for just doing wget the way I had it, and 3 mins 10 seconds for doing wget with the external -i file. – Avid Coder Oct 20 '18 at 3:16
  • After re-reading your initial question, I realized you want no output... did you try doing a wget -q -nv -O /tmp/output.txt -i temp.txt > /dev/null 2>&1 – number9 Oct 20 '18 at 10:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.