1

I am tuning my Elasticsearch cluster, and see that the memory usage doesn't line up. Running "free" on my instance:

free
              total        used        free      shared  buff/cache   available
Mem:       30881324    17072320      238216       49736    13570788    13313528
Swap:             0           0           0

We see that there is 30GB on the machine.

ES was tuned to use 50% of available memory (or max 30GB for oop reasons), and I can verify, here, the -Xms and -Xmx flags are set to 15078m:

/bin/java -Xms15078m -Xmx15078m -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -Des.distribution.flavor=default -Des.distribution.type=rpm -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet

But when running "top" we see a different story.

%MEM is at 59% , many times will show 85% or more (triggering nagios alerts):

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
24366 elastic+  20   0   54.9g  17.5g   1.6g S 267.7 59.3   1849:57 java
1

You have capped the heap size to 15 GB but the JVM is using more than that: stack and native memory, some of which is managed by the JVM (metaspace).

The pmap -x <pid> command might help identifying where the extra 2.5 GB is used, stack, heap.

The ps -o nlwp <pid> command will give you the number of threads. Each thread is consuming 1MB of memory in your case so one thousand threads will use 1 GB.

The jstat -gc <pid> command will show the size of the used metaspace under the MU column.

If the memory is used by the metaspace, you can limit its using this option:

-XX:MaxMetaspaceSize=1g

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.