29

I want to setup an Apache Spark Cluster but I am not able to communicate from the worker machine to the master machine at port 7077 (where the Spark Master is running).

So I tried to telnet to the master from the worker machine and this is what I am seeing:

root@worker:~# telnet spark 7077
Trying 10.xx.xx.xx...
Connected to spark.
Escape character is '^]'.
Connection closed by foreign host.

The command terminated with "Connection closed by foreign host" immediately. It does not timeout or anything.

I verified that the the host is listening on the port and since telnet output shows "Connected to spark." — this also means that the connection is successful.

What could be the reason for such behavior? I am wondering if this closing of the connection could be the reason why I am not able to communicate from my worker machine to the master.

Improve this question
2
  • Is your telnet port (23) allowing access in the iptables on spark? – ryekayo Jul 1 '15 at 18:14
  • iptables on spark is empty. Policy is ACCEPT for INPUT,FORWARD and OUTPUT chains – Mor Eru Jul 1 '15 at 18:23
17

The process that is listening for connections on port 7077 is accepting the connection and then immediately closing the connection. The problem lies somewhere in that application's code or configuration, not in the system itself.

Improve this answer
2
  • So there could be no problem with the connection and maybe this is how the application behaves ? I am new to Apache Spark and I don't know if this is the expected behavior !!! – Mor Eru Jul 1 '15 at 18:24
  • 2
    I doubt this is how the application is supposed to behave. Check the Apache Spark documentation and support sites for configuration debugging information. – John Jul 1 '15 at 18:25
6

I just learned of an odd behaviour in some virtual servers, especially those that are run NAT. You connect to the port exposed by the virtual server to the outside; the server then tries to forward the connection to a target; the target refuses. So the NAT will close your connection with no message whatsoever. Pull your hair out trying to figure out what's going on. This is very much like the previous answer; the source is the interaction in the virtual host itself.

The most likely cause is that someone is using the port, and the application running only accepts one connection at a time, and refuses any additional connections.

Improve this answer
1
  • We've also encountered this error with an external SFTP server. We've inquired with the service provider and they have ruled out any restrictions on ports as well as any other configuration rules that may apply intentionally. We see this problem intermittently on a scheduled job that pulls from the remote host once an hour during business hours. So far they seem to agree with you that the connection is accepted but the virtual host is temporarily denying the connection. – J E Carter II Aug 29 '19 at 12:25
3

The application has policy restrictions to connect only from localhost. Same "problem" with elasticsearch. You can check your app-config or you can make a tunnel. Do

ssh -N -L 7077:127.0.0.1:7077 userxy@spark

then on your machine:

telnet 127.0.0.1 7077
Improve this answer
1

We came across this very error when trying to debug why email (through Python code) sending from a host was failing. It turned out to be due to the mail queue on the mail server being full for this particular host.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.