2

I succeeded in transferring all files from a folder via ftp from a remote server to a raspberry, but I would like to transfer only new ones. Below is the working script i have.

#!/bin/bash -vx
ftp -in IP_SERVER<<END_SCRIPT
quote USER rem_user
quote PASS rem_pass
bin
prompt:off
cd /path_to_server_files
lcd /path_to_local_files
mget *.mp3
bye
END_SCRIPT

I have a company that provides background music to other companies. My method was leaving a computer in each one playing 24/7 or with other specific cron jobs, depending on the client. And the raspberry is a great way to do that instead a computer. The method i have right know that is working is a cron job per folder. Each folder has a type of music. So i will be putting different music from time to time in the server and the cron job will transfer those files once a week. It is set to transfer every mp3 file in that folder to the RPi. The thing is, it will transfer all the files there including the ones that were already there. If i put there, for example, 150 music files, it will take a long time transferring those, not to mention if it is done with all the folders since the RPi ARM is not that powerful. The solution would be not overwriting the files already there, just the new ones. Then after some time another cron job will delete all the files that have more than * days old.

I searched but it seems ftp doesn't have an option like this yet. So I found the wget command which allows to transfer without overwriting but i couldn't make it transfer multiple files. I have been trying to convert the script above with the wget command without success. Can someone with experience in this matter help out? It could be a problem with http also. Thanks in advance.

I have tried with wget command:

* * * * * wget -r -l1 -N -A.mp3 'ftp://serverUser:Password@serverIP/path_to_server_files' /var/www/rd/musica/teste/ftp11.log 2>&1

Errors:

ftp://serverUser:Password@serverIP/path_to_server_files: Bad Port Number

/var/www/rd/musica/teste: Scheme Missing

This is my attempt with rsync:

The rsyncd.conf: (I am not sure if all credentials are right, so I'ĺl put every file in here so it can be corrected.)

lock file = /var/run/rsync.lock
lock file = /var/log/rsyncd.log
pid file = /var/run/rsync.pid

[documents]
  path = /var/www/rd/musica/teste
  comment = The documents folder of localusername
  uid = localusername
  gid = localusername
  read only = no
  list = yes
  auth users = serverusername
  secrets file =/etc/rsync.secrets
  hosts allow = serverIP/255.255.255.0

rsyncd.secrets

localuser:password
serveruser:password

command to run rsync:

rsync -rtv serverusername@serverIP::documents/path_to_server_files/*.mp3 /path_to_local_destination_folder

It returns these errors:

rsync: failed to connect to serverIP (serverIP): Connection refused (111)
rsync error: error in socket IO (code 10) at clientserver.c(122) [Receiver=3.0.9]
  • You might start your question by clearly stating your end-goal, then demonstrate what you've already tried. Fixing your current script may not be the best option, so it's nice to get people on the right mental track early on. – Jeff Schaller Jul 15 '15 at 15:01
  • @Jeff Schaller I don't understand your statement about the end goal, "Tranfering files without overwriting them" is the goal and thats is the question, because in ftp the files are automatically overwritten, and i wanted to change that option so it won't be constantly copying over the same files, just the new ones. As for the examples, the script is one example of what i tried, I can put more, but it might be confusing. I'll try to do it as clear as i can. – angelfmf Jul 15 '15 at 15:20
  • It seems to me that your "core" question is in the last two questions of the third paragraph. Put something like "I am transferring every file now, but I would like to only transfer only new ones" closer to the top of the question instead of burying it. – Jeff Schaller Jul 15 '15 at 15:29
  • Is it important to use FTP rather than (say) rsync? – Toby Speight Jul 15 '15 at 16:22
  • 1
    Also - if you're having trouble with ssh, you can use rsync over its native port since you're clearly not dealing sensitive data. One other comment - watch out for a possible collision of requirements when you want to transfer "files that aren't already there" with "files that have been deleted" -- you'd want to delete (or move) them on the server as well as the Pi so that you don't re-transfer them after the cron job deletes them. – Jeff Schaller Jul 15 '15 at 17:25
1

Have you looked at rsync? From how I interpret your question, this should accomplish exactly what you want. Here is a description how to setup an rsync daemon to connect without ssh.

  • Could you give me an example of a script with rsync together with ftp? I can't seem to find one or making one of my own. Thanks in advance – angelfmf Jul 16 '15 at 11:08
  • @angelmf I added a link with a description. – klimpergeist Jul 16 '15 at 12:15
  • Thank you for the description, but i hasn't work yet. I got a little confused with the credentials as they weren´t very specific in there in wich to put the server and local ones. I've updates the question with what i have done so far – angelfmf Jul 16 '15 at 14:25
  • @angelfmf There is a typo: secrets file =/etc/rsync.secrets instead of rsyncd.secrets – klimpergeist Jul 16 '15 at 17:17
0

You do keep the "old"er files on the source ftp, don't you? If this is the case, the perfect tool to update/sync your local directories would be lftp, which features a mirror command:

lftp -e 'mirror remoteDir localDir;exit' -u 'remoteUser:remotePassword' remoteHost

This would connect to remoteHost as remoteUser using remotePassword, then execute (-e) the command mirror to sync remoteDir to localDir before exiting.

MAN page: http://lftp.yar.ru/lftp-man.html

lftp should be available in the package repositories of all RPi distributions – I myself used it before on an RPi running ArchLinux.

0

SOLUTION - I got this script to work. thank you for all the support you gave, If i have enough time to be working on this i'll keep trying the other options and make them work as well.

#!/usr/bin/python
import os
from ftplib import FTP

local_path='/path_to_local_files/'
os.chdir(local_path)
ftp = FTP(host='server_name_or_IP',user='username', passwd='password')
ftp.cwd('/path_to_local_files/')
f_list = ftp.nlst()
for f in f_list:
if not f.endswith("mp3"):
  continue
new_f_name = local_path + f
if os.path.exists(new_f_name):
  continue
print("Copying remote file <{0}>to local file <{1}>".format(f,new_f_name))
ftp.retrbinary('RETR '+ f, open(new_f_name,'wb').write)

you may need to install this in order for the script to work:

sudo apt-get install python-dev

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.