3

When I run the rsync command to push my files to production, sometimes - not always, and it's very rare - after the sync has completed, we are getting errors within our production environment.

The rsync command is;

rsync -avzO --exclude=*.git --exclude=*.log /var/www/html/ver_1/api root@production.local:/var/www/html/ver_1

I've confirmed the following;

  • Production and test environment settings (PHP and Apache) mirror one another
  • The issue has to be with the rsync command because when I manually copy the files (using WinSCP) there is no issue - so it's not a code fault.
  • We're syncing from a CentOS machine to a CentOS machine (if that makes any difference)

Questions

  • What could be causing the issue?
    • Why would it only happen sometimes (2/3 times out of 25-ish)
  • How can I resolve it, or minimize the occurrence?

Edit

PHP Fatal error: require_once(): Cannot redeclare class api\lib\classapitools\apitools in /var/www/html/ver_1/api/index.php on line 42

  • I've copied the index.php file from test to production, but it didn't fix it.
  • Re-ran the rsync command, and it didn't fix it.
  • Then I copied all the files manually and it worked. I debugged the whole application, but it didn't return anything insightful about the error (I know why the error is there (in a general sense), but the rsync seemed to have FUBAR'd the whole application until a manual sync)
  • What are the "errors" you get in production? – D_Bye Nov 5 '14 at 12:05
  • PHP Fatal error: require_once(): Cannot redeclare class api\lib\classapitools\apitools in /var/www/html/ver_1/api/index.php on line 42 I've copied the index.php file from test to production, but it didn't fix it. Re-ran the rsync command, and it didn't fix it. Then I copied the files manually and it worked. I debugged the whole application, but it didn't return anything insightful about the error (I know why the error is there (in a general sense), but the rsync seemed to have FUBAR'd the whole application until a manual sync) – ʰᵈˑ Nov 5 '14 at 12:12
  • Do you restart the server/app as part of the update procedure? The error message suggests that might be what you're missing, to ensure a clean environment started from the new version of the code. I am not a PHP developer, though, so I may have it wrong... – D_Bye Nov 5 '14 at 12:44
  • We don't restart the server as part of the update procedure as there is no need and the downtime is unwanted - even though it was down for a while, I didn't attempt it. The rsync only updated a small amount of files (although they were core files). I just don't understand why when replacing the whole code manually fixed the issue, and rsync'ing killed it. (You don't compile the PHP application like you would with an app made in C) – ʰᵈˑ Nov 5 '14 at 12:49
  • Try enabling logging in your rsync invocations. Might show up any odd behaviour from rsync. The problem sounds like it might be caused by two processes wanting to access one of the updated files at the same time - the server reading it to serve a request, rsync to write an updated version. You could try using --blocking-io to see if it helps. – D_Bye Nov 5 '14 at 13:06
3

The problem most likely lies in your use of the -a flag on rsync. That is equivalent to a set of other options, including -t, which preserves the modification time of the files.

As you don't restart the application on the target machine, you are now dependent on that machine rereading some or all of those files or not, depending on the application and whether these files are observed as being changed.

You try to use -rlpgoD instead of -a. The "wrong" timestamp of files transferred in the previous rsync (but not in the current), should not cause any troubles, rsync is smart enough to notice that the content is the same even if the timestamps are not.

If you had been using the --in-place option you would probably encounter more problems.

  • So I've ran the rsync scripts a while (from development to test), and using your suggestion the issue hasn't come up again - Thanks! – ʰᵈˑ Nov 6 '14 at 13:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.