0

I need to download some data from some SFTP connection and delete if on that FTP afterwards. But I have to be sure to have downloaded the data correctly.

At the moment I just do this in my script run by a cron:

sftp -oIdentityFile=key user@server <<EOF
    mget -r *
    mdelete *
    bye
EOF

But I worry that if a download breaks because of any reason the files get deleted anyway, since the FTP connection just works the next command without breaking the connection after a failed action.

I need to be sure the files have been downloaded correctly, best would be a hash compare. But since I heavily doubt FTP offers that function, what would be the best solution to solve that problem?

IF NOT possible: It might be a workaround to leave all the files on the FTP server and just download the new ones. This seems to be a way easier solution. Any advice for this solution?

EDIT: Since I have no influence on the server whatsoever, I am searching for the most practicable solution to be sure to download all files before they get deleted. I am not so much worried about the integrity of the data itself (I do trust ssh there).

  • 2
    Hash compare is the way to go, which will require that on the server side a hash file is also written, downloaded, and checked by the client. See RFC 4130 for a standardized version of this process over HTTP. – thrig Dec 28 '17 at 16:03
  • Unfortunately I don't have influence on the served data. – Xali Dec 28 '17 at 16:39
  • The sftp server doesn't provide a hashsum of the file? To what would you compare it to, then? – Jeff Schaller Dec 28 '17 at 16:55
1

Simple sftp scripts like yours are fine for quick-and-dirty fetch scripts, but are a disaster waiting to happen when you use them to also delete files (especially wildcard deletions).

One very obvious danger with your script is that the mdelete * will delete not only the files that were in the directory when you ran mget -r * but also everything that was uploaded to the source directory after you started the mget - even files that weren't included in the mget.

There are several other (safer) alternatives for you. Here's two:

  1. Use sshfs to mount the sftp directory somewhere on your system. Then you can use standard unix commands like cp, mv, rm, rsync etc to move files around. This is by far your best option.

  2. Write a program in a language that has an sftp library (e.g. perl or python. and most common & current compiled languages too) to connect to the remote server and:

    • get a list of files to transfer
    • for each file:
      • download it
      • validate it somehow (e.g. as @thrig suggested with an md5sum file or similar)
      • if it's OK, delete it from the remote server
  • I will write a python script to solve that problem. Thank you – Xali Dec 28 '17 at 16:41
  • i recommend sshfs and rsync. rsync even has a --remove-source-files option to remove files that were successfully transferred. – cas Dec 28 '17 at 16:59
  • Unfortunately I have no control over the server, can't change protocol, can't use the native hashing extension (I assume). – Xali Dec 28 '17 at 17:19
  • you don't need to change the server to use sshfs. the server's already running sftp, you run sshfs on the client side. As far as the server is concerned, it's just another sftp client. – cas Dec 28 '17 at 17:21
  • I just now realized that SFTP is not SSH in FTP, but FTP in SSH, nice! Thank you for that suggestion, would upvote if I wasn't a noob. – Xali Dec 28 '17 at 17:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.