2

First of all: Why I'm trying this? Because we need to download some files AND rename to shorten ones with date "stamps". The remote files have really huge filenames and it's not an option to change(isn't our ftp).

I'm trying to make a bulk download and rename of some files in a remote ftp server, without having to open one ftp connection to each file I have to download. So far, i could achieve renaming and download on-the-fly with nmap ftp command, renaming every file that starts with "N" and ends with ".TXT" to "N_date_time_stamp.TXT"

ftp -niv $url << FTP_COMMAND
        user $user $password
        cd $remotedir
        nmap N*.TXT N_`date "+%H%M%N"`.TXT
        mget N*
        bye
FTP_COMMAND

The problem is: nmap keeps the same %N value to all files passed to mget, and it should change on every download to the current nanosecond value:

250 CWD command successful.
local: N_1054232349627.TXT remote: NO2346662345257245624572457245724562411125555341346134771345123461146-44.TXT
227 Entering Passive Mode (xxxxxxxxxxxxxxx).
125 Data connection already open; Transfer starting.
226 Transfer complete.
2220 bytes received in 0,0995 secs (22 Kbytes/sec)
local: N_1054232349627.TXT remote: NO2346662345257245624572457245724562411125555341346134771345123461146-45.TXT
227 Entering Passive Mode (xxxxxxxxxxxxxx).
125 Data connection already open; Transfer starting.
226 Transfer complete.
2220 bytes received in 0,107 secs (20 Kbytes/sec)

Is there a way to update the nmap on each download?

3

With zsh:

autoload zfinit
zfinit
zmodload zsh/datetime

files=N*.TXT
zfopen $host $user $passwd &&
  zfcd $remotedir &&
  zfrglob files &&
  for f ($files) {
    t=$EPOCHREALTIME
    strftime -s ts %H%M%S ${t%.*}
    ts=$ts.${t#*.}
    zfgcp -Gt $f N_$ts.TXT
  }
zfclose
  • Yeah. I realized that ftp was too much limited and i had to change my script to behave simmilar to yours, but without the zsh :) - Thanks for the info anyway – user34720 Aug 29 '13 at 19:30
3

ftp doesn't strike me as the appropriate tool to do this with. I'd try lftp instead. As you have it scripted the line:

    nmap N*.TXT N_`date "+%H%M%N"`.TXT

Get's evaluated once, and then expanded to the current date, from then on it's only going to be used as the expanded form. What you want to do is get the remote side to do this expansion for you, which I don't believe you can do with regular ftp.

Something like this "might" work through lftp:

$ lftp -u user,password IP <<EOF
!whoami
quit 0
EOF

I do not have access to an FTP server currently so I can't test this. If it works then you may be able to modify it so that the date command runs on the remote server.

References

  • THanks for the help. After reading the docs i found the nlist, that could help me a lot to retrieve the file options without having to wrap lines and colums around with any other software :) – user34720 Aug 29 '13 at 19:29
2

The easy way to do complex things with FTP is to not deal with FTP. Mount the FTP resource as a filesystem. FUSE is widely available nowadays, even on servers with restricted software configurations. You can use Curlftpfs to mount FTP directories.

mkdir /net/someftp.example.com
curlftpfs -o user=bob someftp.example.com /net/someftp.example.com

Store the FTP password in ~/.netrc.

Then use normal shell commands to copy the files with the names you want. I don't recommend this microsecond date scheme as you're dependent on the clock updating fast enough, which could be problematic especially if the clock is adjusted while your script runs. Use a counter instead.

counter=10000
date=$(date +%Y%m%d-%H%M%S)
for x in /net/someftp.example.com/dir/N*.txt; do
  cp -p "$x" "N-$date-${counter#1}.TXT"
  counter=$((counter+1))
done

When you've finished with the FTP directory, unmount it with fusermount -u /net/someftp.example.com.

  • It could be a solution too. :). I just dont know how this would act on a range of 60 ftp servers. I'm explaining just part of the problem here,and the solution that will retrieve those files(EDI related) from a lot of customers(about 60)... If curlftps shows stable on this environment, could simplify a lot our scenario :) – user34720 Aug 30 '13 at 11:21
2

Well, i did some sort of mixed implementation, based on the answers of Stephane and slm. I couldn't use zsh because is a production server and installing a new shell is not an option, so, i used lftp that was installed:

Explanation: On the first here_docs(FTPLISTGET) connect on the ftp server and list the files(nlist). If the listing was successfull( if [ $? -eq 0 ] ) download, one file by one renaming with the current date on the format year,month,day,hour,minute,nanosecond). Some ftps are blazing fast, and saving the second could overwrite the files.

exec_ftp(){

# LIST
LIST_FTP=`lftp $protocol://$url << FTPLISTGET
        user $user $pass
        nlist
        bye
FTPLISTGET`

# Check if list is not empty, proceed...
if [ $? -eq 0 ]; then
        echo "$LIST_FTP" | while read file
        do
                DEST="N_$(date +%Y%m%d%H%M%N).TXT"
                lftp $protocol://$url <<-DOWNLOAD
                        user $user $pass
                        cd $remotedir
                        get $file -o /home/user/$DEST
                        rm $file
                        bye
                DOWNLOAD
        echo "Done in $(date +%d/%m/%Y-%T)" >> /var/log/transfer_ftp.log
        done

# If listing is not possible, 
else
        echo "FTP: $url user: $user - Cant reach host, or wrong credentials" >> /var/log/transfer_ftp_error.log
fi

}

Edit 1: Changed backticks to $(...) as suggested by slm, and added the variable $protocol. Why? Because lftp can download and automate sftp and ftps, and this will be pretty good to us :)

  • Cool solution! One suggestion would be to swap out the backticks (..) and replace them with $(..). The backticks are deprecated. Can you comment on what's going on so that if anyone comes across this in the future it's clear what's going on? A paragraph or 2 would be great. – slm Aug 29 '13 at 19:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy