1

This is really really pain in the ass. I have a ftp server running, and it irregularly generates the latest file. The file is store as:

Home->T22:30:10->new.txt, and the latest one would be (a new folder)

Home->T23:10:25->new.txt (note that this is a new folder with latest time)

I need to implement something (it could be anything, C code, bash script, etc) in a Linux machine that pull the latest file over.

I have look into two options:

Option 1. Use libcurl, pass the directory listing, and select the latest file. This is really pain in the ass and time-consuming and I still can't find a easy way to do this.

Option 2. Use lftp, at initiazation, remove all the files in the server, so that each time when I call lftp to download something, it would be the latest one. (This method is only idealization and I haven't tried in real life).

Is there any easier option?

1

An approach that's often convenient is to mount the files, and then access them like you would access ordinary local files. For a server that you access through FTP, you can use CurlFtpFS.

mkdir theserver
curlftpfs theserver.example.com theserver

You'll need to pass the username and password to curlftpfs, either on the command line (which is unsafe as other users on your machine would be able to see them) or in the file ~/.netrc (strongly recommended). Here's a sample netrc line:

machine theserver.example.com login remoteusername password swordfish

Now that you've mounted the FTP server as a directory on your machine, you can use the usual commands such as ls, cp, etc. For example, to copy the file from the directory that comes last in lexicographic order (should be ok if your file names actually contain the date before the time):

set -- theserver/remote/path/T*
eval "last=\${$#}"
cp -p -- "$last/new.txt" "/some/where/local/${last##*/}.txt"

Or to copy the latest file, assuming that the filenames involved don't contain unprintable characters or newlines:

cd theserver/remote/path
last=$(ls -t -- T*/new.txt | head -n 1)
cp -p -- "$last" "/some/where/local/${last%/*}.txt"
1

If you are OK with running periodically a script, you can utilize wget -N, see:

https://superuser.com/questions/283481/how-do-i-properly-set-wget-to-download-only-new-files


You can create a script like:

cd /var/download_here/
wget -N ftp://example.com/new.txt --ftp-user="my_user" --ftp-password="my_password"

And add crontab entry like:

*/5 *   * * *   /path/to_script

Please take into concern how is the source file created. If something is writing to it and the file is not locked - you can end up in downloading incomplete file.

  • yes, i am ok with that. i will look into more. thank you – andy_ttse Mar 25 '15 at 16:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.