Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I'm working with a cluster of Linux servers sitting behind the Amazon cloud autoscaling feature (documentation). There are files that accumulate on each server, they are cache files for the application. I am searching for an automated way to have the servers sync that folder amongst themselves.

  • Rsync would work if someone had a clever way to do peer detection.
  • Same goes for Unison.
  • We have also considered using a SaaS such as Dropbox.
  • Gluster seems like overkill, and I'm not sure how easy it is to have a new computer auto-join the cluster.

Ideally what would happen is when a new server is launched, it discovers it's peers and begins syncing the folder. From that point on it's a latest filestamp wins P2P synchronization.

Anything out there that does this?

share|improve this question
    
This one doesn't do peer detection, but it may be of interest regardless - mrsync.sourceforge.net. –  Faheem Mitha Apr 8 '11 at 9:44
add comment

5 Answers

Rsync upstream provides support for SLP (Service location protocol). It is not enabled in vanilla rsync sources, but you can find the SLP support in rsync-patches tarball at rsync homepage (slp.diff). E.g. SUSE rsync package is built with this patch, dunno about other distributions.

I'm not sure this is what you are looking for nor do I have any experience with SLP, though.

share|improve this answer
add comment

You know as weird as it might sound but using rtorrent with DHT enabled on the private ec2 range could work

This would give you

  • Automatic Peer discovery
  • Verified file states (yay for hashes)
  • Minimal configuration requirements (add the magnet(or virtual ip for the latest torrent) for the torrent in the start up script)
  • scalability
share|improve this answer
add comment

Maybe I'm missing something here, but is there some reason you don't just setup a simple NFS directory and just have all the machines mount that? Anytime the app writes the new cache file it should just overwrite any old ones.

share|improve this answer
1  
Hi opsguy. What you're missing is what happens when the machine with the NFS fails? We're trying to eliminate single point of failure for the cache. –  user7745 May 25 '11 at 4:04
add comment

I'm pretty sure there is some Amazon management tool, such as Command Line Tools, that permits you to write a boot script to make nodes discover each others, via private or public IP addresses.

By the way, rsync and Unison are not the right tool, if they have to distribute changes to cache very frequently. In this case you should consider some really distributed file system.

share|improve this answer
add comment

Maybe DRBD (distributed block device)?

share|improve this answer
1  
Thanks for your answer, but it lacks substance. DRBD is not so well-known that it doesn't require at least a link. But really, you should give more information: why is DRBD suitable? How would it work? Especially considering that Kevin thought Gluster overkill: why would DRBD not be? –  Gilles Jun 17 '11 at 20:08
    
I'm just realized that you looking for automatic peer discovery, so this one is not suitable for you. DRBD is a block device on your real hardware or on the top of some volume, and than it's sync them over the network. –  MSpike Jun 21 '11 at 8:07
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.