Timeouts between an API and an elasticsearch server every 2 hours

Question

We are experiencing a weird issue on our servers. (Debian 8.9) We have an API which is a PHP application. It requests an elasticsearch which instance is on a separate server.

Every 2 hours, we are experiencing errors 500, it lasts during 1 or 2 minutes, rarely more:

[2017-10-19 20:52:10] +2 hours
[2017-10-19 22:51:59] +2 hours
[2017-10-20 00:52:02] +2 hours
[2017-10-20 02:52:14] +2 hours
[2017-10-20 04:52:28] +2 hours

Sometimes it is +4 hours or +6.

Here is the detail of the error:

request.CRITICAL: Uncaught PHP Exception Elastica\Exception\Connection\HttpException: 
"Operation timed out" 

Which is quite clear. The API tries to connect to the elasticsearch instance until it reaches the specified timeout of the http client.

What could cause this? How to debug this kind of issue?

Of course, when checking later all the URLs referrers, everything is OK.

Answer 1

I would advise raising the somaxconn parameter in the kernel.

Add to /etc/sysctl.conf:

net.core.somaxconn=512

And then run:

sudo sysctl -p

Also in /etc/redis.conf raise tcp-backlog as in (or more):

tcp-backlog 512

From the redis configuration file:

TCP listen() backlog.

In high requests-per-second environments you need an high backlog in order to avoid slow clients connections issues. Note that the Linux kernel will silently truncate it to the value of /proc/sys/net/core/somaxconn so make sure to raise both the value of somaxconn and tcp_max_syn_backlog in order to get the desired effect.

Answer 2

They finally managed to find the issue. And the root cause is totally stupid. If fact a monitoring view of the es cluster sent a lot of queries to es. About 6 times more than the application itself!

As you can see every 2 hours the memory was too high and the server was unavailable several minutes until it clears the memory (garbage collector).

Other parameters were also optimized and/or increased.