2

Configuration : Elasticsearch Version = 5.0.1 OS = Amazon Linux AMI release 2014.03

I am using AWS m3.large instance. I have two elasticsearch nodes. For my data node(node-2) , When I started elasticsearch service with

sudo service elasticsearch start

I found this as the error log.

[2016-11-25T09:31:56,884][INFO ][o.e.b.BootstrapCheck     ] [node-2] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2016-11-25T09:31:56,886][ERROR][o.e.b.Bootstrap          ] [node-2] node validation exception
bootstrap checks failed
max number of threads [1024] for user [elasticsearch] is too low, increase to at least [2048]

Then I edited file /usr/lib/systemd/system/elasticsearch.service and added line

LimitNPROC=2048

When I restarted service. I checked status:

sudo service elasticsearch status

It returned,

elasticsearch dead but subsys locked

In Log

[2016-11-25T09:50:32,138][INFO ][o.e.b.BootstrapCheck     ] [node-2] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2016-11-25T09:50:32,140][ERROR][o.e.b.Bootstrap          ] [node-2] node validation exception
bootstrap checks failed
max number of threads [1024] for user [elasticsearch] is too low, increase to at least [2048]

The output of

[ec2-user@ip-address]$ ls -la /var/lock/subsys
total 8
drwxr-xr-x 2 root root 4096 Nov 25 09:50 .
drwxrwxr-x 5 root lock 4096 Nov 25 03:37 ..
-rw-r--r-- 1 root root    0 Nov  9 19:10 atd
-rw-r--r-- 1 root root    0 Nov  9 19:08 auditd
-rw-r--r-- 1 root root    0 Nov  9 19:08 blk-availability
-rw-r--r-- 1 root root    0 Nov  9 19:10 crond
-rw-r--r-- 1 root root    0 Nov 25 09:50 elasticsearch
-rw-r--r-- 1 root root    0 Nov  9 19:10 local
-rw-r--r-- 1 root root    0 Nov  9 19:07 lvm2-monitor
-rw-r--r-- 1 root root    0 Nov  9 19:08 messagebus
-rw-r--r-- 1 root root    0 Nov  9 19:08 netfs
-rw-r--r-- 1 root root    0 Nov  9 19:08 network
-rw-r--r-- 1 root root    0 Nov  9 19:34 ntpd
-rw-r--r-- 1 root root    0 Nov  9 19:09 ntpdate
-rw------- 1 root root    0 Nov  9 19:08 rsyslog
-rw-r--r-- 1 root root    0 Nov 18 05:07 salt-minion
-rw-r--r-- 1 root root    0 Nov  9 19:10 sendmail
-rw-r--r-- 1 root root    0 Nov  9 19:10 sm-client
-rw-r--r-- 1 root root    0 Nov  9 19:09 sshd

Am I missing something ?

  • Please add to the post ls -la /var/lock/subsys – Rui F Ribeiro Nov 25 '16 at 12:29
  • Please check updated question @RuiFRibeiro – Jakir patel Nov 25 '16 at 13:36
0

You can´t start elasticsearch because a dangling lock file is present in the system.

It is sufficient to delete it with:

sudo rm /var/lock/subsys/elasticsearch

Then you will be able to start elasticsearch again.

If this happens regularly, I would advise investigating the reasons behind it.

  • Thank you for your response. I tried what you suggested and got following output. ` [ec2-user@subsys]$ sudo rm elasticsearch [ec2-user@subsys]$ sudo service elasticsearch status elasticsearch is stopped [ec2-user@subsys]$ sudo service elasticsearch start Starting elasticsearch: [ OK ] [ec2-user@subsys]$ sudo service elasticsearch status elasticsearch (pid 32500) is running... [ec2-user@]$ sudo service elasticsearch status elasticsearch dead but subsys locked ` – Jakir patel Nov 25 '16 at 13:49
  • Apologize for bad formatting @RuiFRiberio. As per your answer I got same output as previous comment. – Jakir patel Nov 25 '16 at 13:55
  • do the * rm*. again, then ulimit -u 2048 . as root, then start it. – Rui F Ribeiro Nov 25 '16 at 14:05
  • I got this output @RuiFRiberio [root@subsys]# rm elasticsearch rm: remove regular empty file ‘elasticsearch’? [root@subsys]# ulimit -u 2048 [root@]# sudo service elasticsearch status elasticsearch dead but subsys locked [root@subsys]# sudo service elasticsearch stop [root@subsys]# sudo service elasticsearch start Starting elasticsearch: [ OK ] [root@subsys]# sudo service elasticsearch status elasticsearch dead but subsys locked [root@subsys]# – Jakir patel Nov 26 '16 at 8:04
  • 1
    I upgraded my Amazon Linux(2014.03) to Amazon linux (2016.09). Now elasticsearch is working fine. – Jakir patel Nov 30 '16 at 10:54
0

I upgraded my Amazon Linux(2014.03) to Amazon Linux(2016.09). Now elasticsearch is working fine.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.