kafka + too many open files between Kafka brokers or may between broker and clients

Question

we are facing very major problem with our Apache Kafka servers our Kafka servers are creased because "Too many open files"

we have production Kafka cluster with 7 machines , while Kafka version is 0.1

OS version - RHEL 7.5

we already increases twice the "kafka_user_nofile_limit" to 500000 , while the original value is 128000

but again we are hit the high value and brokers are crashed

additional we can see clearly that many lines as "Too many open files" are in Kafka logs

we saw the following Jira case

https://issues.apache.org/jira/browse/KAFKA-3317

but we not see what is the solution for this jira or what is the workaround for this major problem ?

from jira

 Bug
Status:OPEN
Priority: Major
Resolution:Unresolved
Affects Version/s:
0.9.0.0, 0.9.0.1
Fix Version/s:None
Component/s:None
Labels:None
Environment:CentOS release 6.5 (Final)
kafka_2.11-0.9.0.1
Flags:
Important
Description
there is much open files in my java app client which write msg to kafka broker, i used

<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>0.9.0.1</version>
</dependency>
new api to write msg. when i use

lsof -p 8780 | wc -l
19184
lsof -p 8780 | grep XmlIpcRegSvc | wc -l
4920
lsof -p 8780 | grep pipe | wc -l
9576
lsof -p 8780 | grep eventpoll | wc -l
4792
where 8780 is my java app pid.

java 37121 app *796u IPv6 960673997 0t0 TCP mad183:50213->mad180:XmlIpcRegSvc (ESTABLISHED)
there are many ESTABLISHED XmlIpcRegSvc , and seems not closed.
and It likes use ipv6 .


 

my configuration

 more /etc/security/limits.d/kafka.conf

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

kafka   - nofile   500000
kafka   - nproc    65536