3

We are experiencing a weird issue on our servers. (Debian 8.9) We have an API which is a PHP application. It requests an elasticsearch which instance is on a separate server.

Every 2 hours, we are experiencing errors 500, it lasts during 1 or 2 minutes, rarely more:

[2017-10-19 20:52:10] +2 hours
[2017-10-19 22:51:59] +2 hours
[2017-10-20 00:52:02] +2 hours
[2017-10-20 02:52:14] +2 hours
[2017-10-20 04:52:28] +2 hours

Sometimes it is +4 hours or +6.

Here is the detail of the error:

request.CRITICAL: Uncaught PHP Exception Elastica\Exception\Connection\HttpException: 
"Operation timed out" 

Which is quite clear. The API tries to connect to the elasticsearch instance until it reaches the specified timeout of the http client.

What could cause this? How to debug this kind of issue?

Of course, when checking later all the URLs referrers, everything is OK.

  • Have you raised somaxconn for redis? net.core.somaxconn=512? – Rui F Ribeiro Oct 23 '17 at 15:53
  • Is this parameter related to Redis? – COil Oct 23 '17 at 16:01
  • I have added an answer. – Rui F Ribeiro Oct 23 '17 at 17:09
2

I would advise raising the somaxconn parameter in the kernel.

Add to /etc/sysctl.conf:

net.core.somaxconn=512

And then run:

sudo sysctl -p

Also in /etc/redis.conf raise tcp-backlog as in (or more):

tcp-backlog 512

From the redis configuration file:

TCP listen() backlog.

In high requests-per-second environments you need an high backlog in order to avoid slow clients connections issues. Note that the Linux kernel will silently truncate it to the value of /proc/sys/net/core/somaxconn so make sure to raise both the value of somaxconn and tcp_max_syn_backlog in order to get the desired effect.

  • Interesting, I will try this. An idea about the recurring 2 hours delay? – COil Oct 23 '17 at 18:08
  • Try it first and see how it goes. – Rui F Ribeiro Oct 23 '17 at 21:05
1

They finally managed to find the issue. And the root cause is totally stupid. If fact a monitoring view of the es cluster sent a lot of queries to es. About 6 times more than the application itself!

es trans log

As you can see every 2 hours the memory was too high and the server was unavailable several minutes until it clears the memory (garbage collector).

Other parameters were also optimized and/or increased.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.