Run LFTP on a list of files

Question

I have a newline-delimited list of paths to files I would like to download via LFTP, from a remote machine to a local machine with the same directory structure. Is there any way that I can pass LFTP the list of files (the entire path to the file on the remote machine), and have it only download those? My current method is individually passing each file to LFTP, downloading it, and then repeating the same process with the next file until my list is exhausted. Obviously, batching the files to download would be much faster, my current solution feels clunky.

Answer 1

How about something like this.

[root@localhost foo]# ls -l file*
-rw-r--r--. 1 root root 33 Jun 30 15:09 filelist
[root@localhost foo]# cat filelist
/tmp/file1
/tmp/file2
/tmp/file3
[root@localhost foo]# awk 'BEGIN { print "open localhost\nuser steve steve\n" } { print "get " $0 } END { print "exit" }' filelist | lftp
[root@localhost foo]# ls -l file*
-rw-r--r--. 1 root root  0 Jun 30 14:57 file1
-rw-r--r--. 1 root root  0 Jun 30 14:57 file2
-rw-r--r--. 1 root root  0 Jun 30 14:57 file3
-rw-r--r--. 1 root root 33 Jun 30 15:09 filelist
[root@localhost foo]#

Answer 2

To expand steve's answer, this script mirrors a list of files if needed while preserving directories.

#!/bin/bash
gawk 'BEGIN { print "open ftp://example.com\n user username password\ncd /remote/dir/\n" } { if (match ($0 ,/.+\//, m)) print "mirror -v -O localbasedir/" m[0] " -f " $0  } END { print "exit" }' filelist | lftp