4

I want to write a lftp script that will download files every 15 minutes from a server every x amount of time.

Can someone advise how i can do this?

Thanks

9

First: Create a script. You can call it whatever you want. I will call it downloader.sh.

#!/bin/bash
PROTOCOL="ftp"
URL="server.example.com" 
LOCALDIR="/home/user/downloads"
REMOTEDIR="dir/remote/server/"
USER="user"
PASS="password"
REGEX="*.txt"
LOG="/home/user/script.log"

cd $LOCALDIR
if [  ! $? -eq 0 ]; then
    echo "$(date "+%d/%m/%Y-%T") Cant cd to $LOCALDIR. Please make sure this local directory is valid" >> $LOG
fi

lftp  $PROTOCOL://$URL <<- DOWNLOAD
    user $USER "$PASS"
    cd $REMOTEDIR
    mget -E $REGEX
DOWNLOAD

if [ ! $? -eq 0 ]; then
    echo "$(date "+%d/%m/%Y-%T") Cant download files. Make sure the credentials and server information are correct" >> $LOG
fi

Second: Add it to crontab. If you want to execute it every exact 15 minutes inside an hour:

45,30,15,00 * * * * /home/user/downloader.sh >/dev/null 2>&1

If you want to execute it each 15 minutes no matter what is the starting minute:

*/15 * * * * /home/user/downloader.sh >/dev/null 2>&1

Explaining the variables:

  • PROTOCOL - What protocol to use. lftp supports a good range of them: ftp, ftps, http, https, hftp, fish, sftp and file. https and ftps require lftp to be compiled with OpenSSL or GNU TLS support.
  • URL- Name or IP of the server. You can even add :PORT at the end if your server doesn't use the default port of the protocol being used.
  • LOCALDIR - Where to save the files.
  • REMOTEDIR - Where to cd on the remote server to get the files.
  • USER and PASSWORD - ftp credentials.
  • REGEX - Regular expression to filter files to download. It can be useful if you want to download only files of a determined extension, for example. Use * if you want to download everything.
  • LOG - Logfile location.

Explaining some code logic:

1. - if

if [  ! $? -eq 0 ]; then
fi

The $? variable is a special bash variable that means "status code of last command". Bash always return zero on successful command executions so, comparing -eq (equal to) with the starting ! (negative) on an if should be enough to see if cd and lftp had issues during execution. If you want a better log of what happened, you will have to crawl through those commands' documentation.

2. - heredocs

lftp  $PROTOCOL://$URL <<- DOWNLOAD
DOWNLOAD

bash heredocs. It's a way to say "feed this command with this input list". I've named the limit string DOWNLOAD so, everything between <<- DOWNLOAD and DOWNLOAD will be input to lftp. You will see examples on the internet with << symbol but I prefer the <<- version since it supports indentation.

3. - lftp commands

    user $USER "$PASS"
    cd $REMOTEDIR
    mget -E $REGEX

These are internal commands of lftp that means respectively, auth the user with $USER login and "$PASS" password, change to $REMOTEDIR and bulk download anything with the $REGEX keywords. You can learn them by simply typing lftp, and as soon as an lftp shell is opened, type ? and press Enter or ? lftp-command-you-want and press Enter. Example:

[root@host ~]# lftp
lftp :~> ?
    !<shell-command>                     (commands)                           alias [<name> [<value>]]
    attach [PID]                         bookmark [SUBCMD]                    cache [SUBCMD]
    cat [-b] <files>                     cd <rdir>                            chmod [OPTS] mode file...
    close [-a]                           [re]cls [opts] [path/][pattern]      debug [<level>|off] [-o <file>]
    du [options] <dirs>                  exit [<code>|bg]                     get [OPTS] <rfile> [-o <lfile>]
    glob [OPTS] <cmd> <args>             help [<cmd>]                         history -w file|-r file|-c|-l [cnt]
    jobs [-v] [<job_no...>]              kill all|<job_no>                    lcd <ldir>
    lftp [OPTS] <site>                   ln [-s] <file1> <file2>              ls [<args>]
    mget [OPTS] <files>                  mirror [OPTS] [remote [local]]       mkdir [-p] <dirs>
    module name [args]                   more <files>                         mput [OPTS] <files>
    mrm <files>                          mv <file1> <file2>                   [re]nlist [<args>]
    open [OPTS] <site>                   pget [OPTS] <rfile> [-o <lfile>]     put [OPTS] <lfile> [-o <rfile>]
    pwd [-p]                             queue [OPTS] [<cmd>]                 quote <cmd>
    repeat [OPTS] [delay] [command]      rm [-r] [-f] <files>                 rmdir [-f] <dirs>
    scache [<session_no>]                set [OPT] [<var> [<val>]]            site <site-cmd>
    source <file>                        torrent [-O <dir>] <file|URL>...     user <user|URL> [<pass>]
    wait [<jobno>]                       zcat <files>                         zmore <files>

lftp :~> ? mget
Usage: mget [OPTS] <files>
Gets selected files with expanded wildcards
 -c  continue, resume transfer
 -d  create directories the same as in file names and get the
     files into them instead of current directory
 -E  delete remote files after successful transfer
 -a  use ascii mode (binary is the default)
 -O <base> specifies base directory or URL where files should be placed

The knowledge related to know that mget would be the right command inside lftp came from reading manpages and searching for keywords like "bulk", "multi" or "mass", and knowing that the ftp(1) command also have the mget command so, probably lftp would have an equivalent.

Manpage: lftp(1)

  • Thanks for this. Is there any way i can log to a .txt file what is happening when this runs? – gurpsone Jan 12 '16 at 13:29
  • sure - either log it internally to the script, or change the tail end of the cron job from ">/dev/null 2>&1" to "> /some/log/file 2>&1" (or separately pipe stderr elsewhere) – Jeff Schaller Jan 12 '16 at 13:34
  • Since i did this script in a rush, i will add some logging inside of it. – user34720 Jan 12 '16 at 15:32
  • @nwildner I've been looking around the internet to see what the -dt operator means in your control-flow conditions, but I can't find an answer. Care to explain? – Black Milk May 8 '17 at 19:20
  • @nwildner: I’ve never heard of a -dt operator in bash.  My copy doesn’t recognize it.  But, most importantly, the document you linked to doesn’t mention it. Are you using a non-English version of bash?  Are you translating the “different than” operator from your version of bash into English (manually, rather than checking what the English version of bash actually supports)? – G-Man Says 'Reinstate Monica' Apr 9 '18 at 1:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.