I know what you are thinking right now "Just use rsync" but unfortunately this system does not have rsync, and I would like to come up with another solution.

The Setup:

  1. The source Server has a Upload folder containing 140 000+ Files (No Sub Directories)
  2. The destination Server needs the Upload folder containing 140 000+ Files

Let's call it Migrating with only Post-It Notes... Now to complicate matters, Server 1 keeps getting new files every day, due to new Uploads or Thumbnails being generated, so that idea in your head right now "just copy the file from 1 to 2" will not work, as once I am done with the Copy of about 20+GB I can start again, as there is already new file on the source Server...

My Solution Idea

  • Copy the complete folder from source to destination
  • Find the latest create date in the destination Server and use as starting point on the source Server
  • Copy all files since the last create date from the source to the destination (delta)
  • Setup a cron to do this as often as possible.

My Problem

find /uploads/* -mtime -1 bash: /bin/find: Argument list too long

Now before I start writing a bash script loop , I was wondering if there is someone out there that could suggest another way of doing this without a bash script... good old low level CLI.

  • right now I am using a 3rd Server, to Copy the Files once a day, and then do a Rsync to the Destination, but this feels like a Windows administrator solution.
    – Adesso
    Aug 31 '16 at 11:37
  • 3
    Remove that star in /uploads/*.
    – Kusalananda
    Aug 31 '16 at 11:44
  • Can you either install software on the destination server or mount the source and target directories on a third machine?
    – StrongBad
    Aug 31 '16 at 11:48
  • 2
    Just use rsync or some other similar tools. You'll end up re-implementing what they've done already, usually badly, and you won't notice the bugs in your setup until after you've hit issues/customer complaints/whatever. Don't waste your time.
    – Mat
    Aug 31 '16 at 11:55
  • 2
    will cp -u work? "copy only when the SOURCE file is newer than the destination file or when the destination file is missing"
    – Sundeep
    Aug 31 '16 at 12:13

I have managed to find a solution, but not in a single line yet. I am still working on that.

Assuming that you can trust the Filesystem Create/Modified Date, one can use find to generate a list of files, output this in a text file, and then with a loop do a scp to the destination Server. I can't seem to get the find result directly to scp correctly.

I used a helper Programm sshpass, but it would be better to use keyfiles, I know

find files in source Server from the destination Server and save the result limiting the result to only one directory and only files.

sshpass -p 'MyPassword' ssh [user@server] "find [path/to/look/in] -maxdepth 1 -mtime -1 -type f" > /tmp/last-24-hours-file-list.txt

Loop the output file and invoke scp per file

while read filename; do sshpass -p 'MyPassword' scp -Crpv [user@server]:$filename [path/to/save/to]; done < /tmp/last-24-hours-file-list.txt
  • 1
    find has an -exec option that executes a command per file, this will make your script much more robust to weird filenames. You really should use asymmetric keys rather than write your password directly in plain text. Also, for the "older" files... from the description you gave it sounds like an md5checksum would make sense. Good luck!
    – RSFalcon7
    Sep 5 '16 at 13:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.