7

I have a newline-delimited list of paths to files I would like to download via LFTP, from a remote machine to a local machine with the same directory structure. Is there any way that I can pass LFTP the list of files (the entire path to the file on the remote machine), and have it only download those? My current method is individually passing each file to LFTP, downloading it, and then repeating the same process with the next file until my list is exhausted. Obviously, batching the files to download would be much faster, my current solution feels clunky.

3

How about something like this.

[root@localhost foo]# ls -l file*
-rw-r--r--. 1 root root 33 Jun 30 15:09 filelist
[root@localhost foo]# cat filelist
/tmp/file1
/tmp/file2
/tmp/file3
[root@localhost foo]# awk 'BEGIN { print "open localhost\nuser steve steve\n" } { print "get " $0 } END { print "exit" }' filelist | lftp
[root@localhost foo]# ls -l file*
-rw-r--r--. 1 root root  0 Jun 30 14:57 file1
-rw-r--r--. 1 root root  0 Jun 30 14:57 file2
-rw-r--r--. 1 root root  0 Jun 30 14:57 file3
-rw-r--r--. 1 root root 33 Jun 30 15:09 filelist
[root@localhost foo]#
2

To expand steve's answer, this script mirrors a list of files if needed while preserving directories.

#!/bin/bash
gawk 'BEGIN { print "open ftp://example.com\n user username password\ncd /remote/dir/\n" } { if (match ($0 ,/.+\//, m)) print "mirror -v -O localbasedir/" m[0] " -f " $0  } END { print "exit" }' filelist | lftp
  • Any chance you could give some more detail about that GAWK string? – Robert Mark Bram Aug 3 '16 at 6:06
  • lftp mirror can only mirror folders, so I take a list of paths, extract the folder from them, then construct a lftp script with mirror commands for each of the files with their folders ... this means lftp will only copy them from the ftp if they have changed since the last sync – MarZab Aug 4 '16 at 5:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.