We are experiencing a weird issue on our servers. (Debian 8.9) We have an API which is a PHP application. It requests an elasticsearch which instance is on a separate server.
Every 2 hours, we are experiencing errors 500, it lasts during 1 or 2 minutes, rarely more:
[2017-10-19 20:52:10] +2 hours [2017-10-19 22:51:59] +2 hours [2017-10-20 00:52:02] +2 hours [2017-10-20 02:52:14] +2 hours [2017-10-20 04:52:28] +2 hours
Sometimes it is +4 hours or +6.
Here is the detail of the error:
request.CRITICAL: Uncaught PHP Exception Elastica\Exception\Connection\HttpException: "Operation timed out"
Which is quite clear. The API tries to connect to the elasticsearch instance until it reaches the specified timeout of the http client.
What could cause this? How to debug this kind of issue?
Of course, when checking later all the URLs referrers, everything is OK.