Sync two Directories without rsync

Question

I know what you are thinking right now "Just use rsync" but unfortunately this system does not have rsync, and I would like to come up with another solution.

The Setup:

1. The source Server has a Upload folder containing 140 000+ Files (No Sub Directories)
2. The destination Server needs the Upload folder containing 140 000+ Files

Let's call it Migrating with only Post-It Notes... Now to complicate matters, Server 1 keeps getting new files every day, due to new Uploads or Thumbnails being generated, so that idea in your head right now "just copy the file from 1 to 2" will not work, as once I am done with the Copy of about 20+GB I can start again, as there is already new file on the source Server...

My Solution Idea

• Copy the complete folder from source to destination
• Find the latest create date in the destination Server and use as starting point on the source Server
• Copy all files since the last create date from the source to the destination (delta)
• Setup a cron to do this as often as possible.

My Problem

find /uploads/* -mtime -1 bash: /bin/find: Argument list too long

Now before I start writing a bash script loop , I was wondering if there is someone out there that could suggest another way of doing this without a bash script... good old low level CLI.

Answer 1

I have managed to find a solution, but not in a single line yet. I am still working on that.

Assuming that you can trust the Filesystem Create/Modified Date, one can use find to generate a list of files, output this in a text file, and then with a loop do a scp to the destination Server. I can't seem to get the find result directly to scp correctly.

I used a helper Programm sshpass, but it would be better to use keyfiles, I know

find files in source Server from the destination Server and save the result limiting the result to only one directory and only files.

sshpass -p 'MyPassword' ssh [user@server] "find [path/to/look/in] -maxdepth 1 -mtime -1 -type f" > /tmp/last-24-hours-file-list.txt

Loop the output file and invoke scp per file

while read filename; do sshpass -p 'MyPassword' scp -Crpv [user@server]:$filename [path/to/save/to]; done < /tmp/last-24-hours-file-list.txt