1

I know what you are thinking right now "Just use rsync" but unfortunately this system does not have rsync, and I would like to come up with another solution.

The Setup:

  1. The source Server has a Upload folder containing 140 000+ Files (No Sub Directories)
  2. The destination Server needs the Upload folder containing 140 000+ Files

Let's call it Migrating with only Post-It Notes... Now to complicate matters, Server 1 keeps getting new files every day, due to new Uploads or Thumbnails being generated, so that idea in your head right now "just copy the file from 1 to 2" will not work, as once I am done with the Copy of about 20+GB I can start again, as there is already new file on the source Server...

My Solution Idea

  • Copy the complete folder from source to destination
  • Find the latest create date in the destination Server and use as starting point on the source Server
  • Copy all files since the last create date from the source to the destination (delta)
  • Setup a cron to do this as often as possible.

My Problem

find /uploads/* -mtime -1 bash: /bin/find: Argument list too long

Now before I start writing a bash script loop , I was wondering if there is someone out there that could suggest another way of doing this without a bash script... good old low level CLI.

6
  • right now I am using a 3rd Server, to Copy the Files once a day, and then do a Rsync to the Destination, but this feels like a Windows administrator solution.
    – Adesso
    Commented Aug 31, 2016 at 11:37
  • 5
    Remove that star in /uploads/*.
    – Kusalananda
    Commented Aug 31, 2016 at 11:44
  • Can you either install software on the destination server or mount the source and target directories on a third machine?
    – StrongBad
    Commented Aug 31, 2016 at 11:48
  • 4
    Just use rsync or some other similar tools. You'll end up re-implementing what they've done already, usually badly, and you won't notice the bugs in your setup until after you've hit issues/customer complaints/whatever. Don't waste your time.
    – Mat
    Commented Aug 31, 2016 at 11:55
  • 3
    will cp -u work? "copy only when the SOURCE file is newer than the destination file or when the destination file is missing"
    – Sundeep
    Commented Aug 31, 2016 at 12:13

1 Answer 1

0

I have managed to find a solution, but not in a single line yet. I am still working on that.

Assuming that you can trust the Filesystem Create/Modified Date, one can use find to generate a list of files, output this in a text file, and then with a loop do a scp to the destination Server. I can't seem to get the find result directly to scp correctly.

I used a helper Programm sshpass, but it would be better to use keyfiles, I know

find files in source Server from the destination Server and save the result limiting the result to only one directory and only files.

sshpass -p 'MyPassword' ssh [user@server] "find [path/to/look/in] -maxdepth 1 -mtime -1 -type f" > /tmp/last-24-hours-file-list.txt

Loop the output file and invoke scp per file

while read filename; do sshpass -p 'MyPassword' scp -Crpv [user@server]:$filename [path/to/save/to]; done < /tmp/last-24-hours-file-list.txt
3
  • 2
    find has an -exec option that executes a command per file, this will make your script much more robust to weird filenames. You really should use asymmetric keys rather than write your password directly in plain text. Also, for the "older" files... from the description you gave it sounds like an md5checksum would make sense. Good luck!
    – RSFalcon7
    Commented Sep 5, 2016 at 13:16
  • Something similar was my first idea before I saw your answer. Assuming you have md5sum on both computers, I would make md5sum list of all files. Then compare the lists with diff, and copy only the ones that have different md5 sums, or are missing in the destination list. You solution is OK, too.
    – nobody
    Commented Aug 21, 2023 at 6:59
  • So what you changed was going from find /path/* to find /path/. As a solution it still sucks. You've not really provided any explanation why you can't use rsync. You might be able to track the changes with inotify (assuming you can't instrument the code which is putting files in there). Using -mtime -1 is crude - does your version of find not support -cnewer ?
    – symcbean
    Commented Aug 21, 2023 at 10:59

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .