I am working with a embedded linux unit, here's my situation:

I use my computer(Ubuntu 16.04 machine) to telnet onto a unit issuing reboot command, then the unit reboots successfully.

Here comes the problem, I expect the telnet will return and prompt Connection closed by foreign host. when the unit shuts itself down. But it doesn't in time, on the contrary, it prompts Connection closed by foreign host. only after

  1. I hit return key from my computer AND

  2. after/during the unit booting

When I do the same thing to other unit from my computer, there's no such problem.

Why is that? Is this a problem of telnet, my computer or the embedded unit? How can I fix it?

  • Are the two units running identical operating systems? – Mark Plotnick May 10 '18 at 19:04
  • @MarkPlotnick Yes. Hardware is a little different though, the problem unit has no Bluetooth, the good unit has it. And some other minor hardware differences.. – Henry May 10 '18 at 19:23
  • Ordinarily, when the telnet server exits or is killed, the TCP connection is closed cleanly and the client politely gets sent a FIN packet. But if the server shuts down without killing off individual processes, the TCP connections aren't closed. (Same thing if the server crashes/panics). When the server comes back up, then when the client sends a packet to it (because you typed something), the server sends the client an RST packet because it doesn't recognize this packet as a part of a valid TCP connection. The below answers give you some workarounds. – Mark Plotnick May 10 '18 at 21:00
  • @MarkPlotnick that sounds reasonable but this same problem happens every time, not sure if race condition is the cause – Henry May 10 '18 at 21:01

Most likely, your client system (the one you're running telnet from) does not have TCP keepalive enabled globally (and it arguably should not), and the embedded system isn't closing the connection gracefully.

This is a common problem with network services that spawn separate processes for each user session, and then don't terminate them when the main process gets shut down (on the other side of the coin though, you can restart the service for an upgrade without affecting connected users, which is why they do this in the first place).

  • Thanks for your answer, Austin! I checked TCP Keepalive parameters on my computer they are all set to standard values on ubuntu, which are 7200(tcp_keepalive_time), 75(tcp_keepalive_intvl), 9(tcp_keepalive_probes) and I would like to keep it that way, just dont touch it since only this unit has this problem. So do you know how to make the embedded linux close the connection gracefully? – Henry May 10 '18 at 20:29

You can enable keepalive for just your telnet connection by applying some options on the socket. Rather than code this and recompile the program, you can simply use an existing library libkeepalive.

Download the source tar file, libkeepalive-0.3.tar.gz, and compile the library with the command make.

tar xvzf libkeepalive-0.3.tar.gz
cd libkeepalive-0.3

You will have a file libkeepalive.so. You can now run your telnet with the library pre-loaded. It will do the appropriate changes to the socket() function. You have 3 values you can set in the environment, as for the global keepalive. For example,

LD_PRELOAD=$PWD/libkeepalive.so \
 KEEPCNT=2 KEEPIDLE=2 KEEPINTVL=4 telnet hostname

Now, when the remote stops responding to the keepalive packets, the telnet command will exit with Connection closed by foreign host.

See the README file in the sources:

KEEPCNT     <=>   net.ipv4.tcp_keepalive_probes
KEEPIDLE    <=>   net.ipv4.tcp_keepalive_time
KEEPINTVL   <=>   net.ipv4.tcp_keepalive_intvl

After KEEPIDLE seconds a keepalive packet will be sent every KEEPINTVL seconds. After KEEPCNT packets with no reply, disconnect.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.