I'm debugging Hadoop DataNodes that won't start. We are using saltstack and also elasticsearch on the machines.
The Hadoop DataNode error is pretty clear:
java.net.BindException: Problem binding to [0.0.0.0:50020] java.net.BindException: Address already in use; For more details see: http://wiki.apache.org/hadoop/BindException [...] Caused by: java.net.BindException: Address already in use [...] (ExitUtil.java:terminate(124)) - Exiting with status 1
lsof -i -n for port 50020 says it's already used but only as source port and not destination port:
salt-mini 1733 root 25u IPv4 17452 0t0 TCP xx.xx.132.72:50020->xx.xx.132.20:4505 (ESTABLISHED) java 2789 elasticsearch 2127u IPv6 9808 0t0 TCP xx.xx.132.72:50020->xx.xx.132.55:9300 (ESTABLISHED)
However binding on 0.0.0.0 does not seem to work:
root@host:~# nc -l 50020 nc: Address already in use
Is this intentional? Is binding to 0.0.0.0 disallowed when the port is already used a source port? There is nothing listening to the socket - I don't really know why it shouldn't work.
root@host:~# uname -a Linux host 4.2.0-19-generic #23~14.04.1-Ubuntu SMP Thu Nov 12 12:33:30 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux