telnet to an host/port works while no service on this host listening on this port

Question

I am crashing my head against the wall with this one...

I run nmap from my desktop for a specific server I have, to check some opened ports which could be security issues. It returns ports which shouldn't be opened :

nmap -T5 WWW.XXX.YYY.ZZZ
Starting Nmap 7.12 ( https://nmap.org ) at 2016-08-09 21:21 CEST
Nmap scan report for mydomain.com (WWW.XXX.YYY.ZZZ)
Host is up (0.0034s latency).
Not shown: 992 filtered ports
PORT     STATE  SERVICE
80/tcp   open   http
110/tcp  open   pop3
143/tcp  open   imap
443/tcp  open   https
993/tcp  open   imaps
995/tcp  open   pop3s

Nmap done: 1 IP address (1 host up) scanned in 3.33 seconds 

http and https have to be opened. But not the other ones... I run this test from 5 different IP addresses.

Ok maybe it's false positives due to how nmap select open/filtered/closed ports. So I tried to telnet those ports and... I do have a connexion :

[meandmyself@MacBook ~]$ telnet WWW.XXX.YYY.ZZZ 110
Trying WWW.XXX.YYY.ZZZ...
Connected to mydomain.com.
Escape character is '^]'.

On the target host, I double checked my netstat and iptables to see if I missed something :

[21:55] ovh-user@mydomain:~ $ sudo netstat -tanlp | grep LISTEN
[sudo] password for ovh-user:
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1016/master
tcp        0      0 WWW.XXX.YYY.ZZZ:4444      0.0.0.0:*               LISTEN      353/sshd
tcp        0      0 0.0.0.0:5665            0.0.0.0:*               LISTEN      2485/icinga2
tcp        0      0 127.0.0.1:8999          0.0.0.0:*               LISTEN      826/php-fpm.conf)
tcp        0      0 0.0.0.0:3306            0.0.0.0:*               LISTEN      25433/mysqld
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      346/rpcbind
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      828/nginx -g daemon
tcp6       0      0 :::9000                 :::*                    LISTEN      25477/java
tcp6       0      0 :::111                  :::*                    LISTEN      346/rpcbind

Except for rpcbind, i know all these services, only nginx is reachable from outside. Ther is nothing on ports 110, 143, 993 and 995...

Then I double checked my iptables :

[21:33] ovh-user@mydomain:~ $ sudo iptables -v -L INPUT
Chain INPUT (policy DROP 8 packets, 352 bytes)
 pkts bytes target     prot opt in     out     source               destination
 2602  749K ACCEPT     all  --  any    any     anywhere             anywhere             state RELATED,ESTABLISHED
   76  5472 ACCEPT     all  --  lo     any     anywhere             anywhere
  272  8912 ACCEPT     icmp --  any    any     anywhere             anywhere
    1    64 ACCEPT     tcp  --  any    any     anywhere             anywhere             tcp dpt:4444
   27  1604 ACCEPT     tcp  --  any    any     anywhere             anywhere             tcp dpt:http
    1    44 ACCEPT     tcp  --  any    any     anywhere             anywhere             tcp dpt:https
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             tcp spt:ftp state ESTABLISHED
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             tcp spt:ftp-data state RELATED,ESTABLISHED
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             tcp spts:1024:65535 dpts:1024:65535 state ESTABLISHED

[21:34] ovh-user@mydomain:~ $ sudo iptables -v -L FORWARD
Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
[21:55] ovh-user@mydomain:~ $ sudo iptables -t nat -nvL PREROUTING
Chain PREROUTING (policy ACCEPT 1342 packets, 49898 bytes)
 pkts bytes target     prot opt in     out     source               destination

There is nothing opened on pop or imap ports...

From my basic understanding of unix services and firewalling I don't get it. Why does telnet gives me "Connected to mydomain.com" for a port with no process listening on it ? Is there a way to trace how telnet reaches something ?

Answer 1

A detailed answer for those crashing their head against the wall too.

1) Find the opened ports of your server

First, use nmap to diagnose which ports are opened and which ones are closed / filetered. Nmap has different scanning modes to find the open ports, you should try all these modes.

You end up with a list of opened ports, some of them are logical but others no.

In my case, ports 110, 143, 993, 995 were not supposed to be opened.

2) Are these ports really opened

Use telnet or nc to test if this TCP ports are really opened

[meandmyself@MacBook ~]$ telnet WWW.XXX.YYY.ZZZ 110
Trying WWW.XXX.YYY.ZZZ...
Connected to mydomain.com.
Escape character is '^]'.

In my case there is a real answer when I try to reach port 110 of my server.

Here is the scary part, the port seems to be opened.

3) Diagnose on the server

A) Is there a running process on my server listening on this port

netstat or lsof can show you if there are running processes listening on these opened ports.

[21:55] ovh-user@mydomain:~ $ sudo netstat -tanlp | grep LISTEN
[sudo] password for ovh-user:
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1016/master
tcp        0      0 WWW.XXX.YYY.ZZZ:4444      0.0.0.0:*               LISTEN      353/sshd
tcp        0      0 0.0.0.0:5665            0.0.0.0:*               LISTEN      2485/icinga2
tcp        0      0 127.0.0.1:8999          0.0.0.0:*               LISTEN      826/php-fpm.conf)
tcp        0      0 0.0.0.0:3306            0.0.0.0:*               LISTEN      25433/mysqld
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      346/rpcbind
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      828/nginx -g daemon
tcp6       0      0 :::9000                 :::*                    LISTEN      25477/java
tcp6       0      0 :::111                  :::*                    LISTEN      346/rpcbind

In my case there is no process listening on port 110, 993... Just processes listening on ports 25, 4444, 5665, 8999, 3306, 111, 80, 9000.

B) Is my server allowing incoming requests to port 110

Diagnose the iptables rules for INPUT, FORWARD and PREROUTING chains

[21:33] ovh-user@mydomain:~ $ sudo iptables -v -L INPUT

In my case there is a default DROP policy, which means that every open ports have to be clearly specified to have a chance to reach a running process.

I can now conclude that there is no chance that my server answers for request on port 110.

4) Who is answering to requests on port 110 to my server ?

Every actif nodes in the path between my computer and my server could be the one which answers the requests. Keep in mind that it also includes my own computer.

To diagnose this, test with nmap and telnet from :

• A different computer
• A different access point / ISP box
• A different location (use ssh tunnels and other servers to proxy your requests)

In my case I tested with another server located in the same datacenter. The point was to see if a front firewall, set up by my hosting provider, could handle requests on port 110, 143... I also tested telnet on port 110 with my phone and JuiceSSh over 4G.

And there wasn't any answer for requests on port 110 with these tests.

5) So it's my computer which opens these ports locally

In my case I was testing from OSX 10.11.5. I run lsof and netstat to track if my own computer listen on specific ports :

[meandmyself@MacBook ~]$ sudo netstat -anv | grep LISTEN
tcp46      0      0  *.3306                 *.*                    LISTEN      131072 131072    827      0
tcp4       0      0  127.0.0.1.4380         *.*                    LISTEN      131072 131072    453      0
tcp4       0      0  127.0.0.1.4370         *.*                    LISTEN      131072 131072    453      0
tcp6       0      0  ::1.12993              *.*                    LISTEN      131072 131072    352      0
tcp4       0      0  127.0.0.1.12993        *.*                    LISTEN      131072 131072    352      0
tcp6       0      0  ::1.12995              *.*                    LISTEN      131072 131072    352      0
tcp4       0      0  127.0.0.1.12995        *.*                    LISTEN      131072 131072    352      0
tcp6       0      0  ::1.12143              *.*                    LISTEN      131072 131072    352      0
tcp4       0      0  127.0.0.1.12143        *.*                    LISTEN      131072 131072    352      0
tcp6       0      0  ::1.12110              *.*                    LISTEN      131072 131072    352      0
tcp4       0      0  127.0.0.1.12110        *.*                    LISTEN      131072 131072    352      0
tcp6       0      0  ::1.12443              *.*                    LISTEN      131072 131072    352      0
tcp4       0      0  127.0.0.1.12443        *.*                    LISTEN      131072 131072    352      0
tcp6       0      0  ::1.12080              *.*                    LISTEN      131072 131072    352      0
tcp4       0      0  127.0.0.1.12080        *.*                    LISTEN      131072 131072    352      0
tcp46      0      0  *.80                   *.*                    LISTEN      131072 131072    321      0
tcp4       0      0  127.0.0.1.49153        *.*                    LISTEN      131072 131072     78      0
tcp4       0      0  127.0.0.1.49152        *.*                    LISTEN      131072 131072     78      0

[meandmyself@MacBook ~]$ sudo lsof -i -n -P |grep LISTEN
mtmfs       78           root    4u  IPv4 0x7f85fc078645d63b      0t0    TCP 127.0.0.1:49152 (LISTEN)
mtmfs       78           root    6u  IPv4 0x7f85fc078645cd43      0t0    TCP 127.0.0.1:49153 (LISTEN)
httpd       83           root    4u  IPv6 0x7f85fc0788d37b8b      0t0    TCP *:80 (LISTEN)
httpd      321           _www    4u  IPv6 0x7f85fc0788d37b8b      0t0    TCP *:80 (LISTEN)
com.avast  352           root    3u  IPv4 0x7f85fc0788dc363b      0t0    TCP 127.0.0.1:12080 (LISTEN)
com.avast  352           root    4u  IPv6 0x7f85fc0788d380eb      0t0    TCP [::1]:12080 (LISTEN)
com.avast  352           root    5u  IPv4 0x7f85fc078645df33      0t0    TCP 127.0.0.1:12443 (LISTEN)
com.avast  352           root    6u  IPv6 0x7f85fc0788d3864b      0t0    TCP [::1]:12443 (LISTEN)
com.avast  352           root    7u  IPv4 0x7f85fc0788dc3f33      0t0    TCP 127.0.0.1:12110 (LISTEN)
com.avast  352           root    8u  IPv6 0x7f85fc0788d38bab      0t0    TCP [::1]:12110 (LISTEN)
com.avast  352           root    9u  IPv4 0x7f85fc0788dc2d43      0t0    TCP 127.0.0.1:12143 (LISTEN)
com.avast  352           root   10u  IPv6 0x7f85fc0788d3910b      0t0    TCP [::1]:12143 (LISTEN)
com.avast  352           root   11u  IPv4 0x7f85fc0788dc482b      0t0    TCP 127.0.0.1:12995 (LISTEN)
com.avast  352           root   12u  IPv6 0x7f85fc0788d3762b      0t0    TCP [::1]:12995 (LISTEN)
com.avast  352           root   13u  IPv4 0x7f85fc0788dc5123      0t0    TCP 127.0.0.1:12993 (LISTEN)
com.avast  352           root   14u  IPv6 0x7f85fc0788d370cb      0t0    TCP [::1]:12993 (LISTEN)
SpotifyWe  453        kheraud    6u  IPv4 0x7f85fc078a53282b      0t0    TCP 127.0.0.1:4370 (LISTEN)
SpotifyWe  453        kheraud    7u  IPv4 0x7f85fc078a533123      0t0    TCP 127.0.0.1:4380 (LISTEN)
mysqld     827         _mysql   33u  IPv6 0x7f85fc0788d36b6b      0t0    TCP *:3306 (LISTEN)

Lsof gives me a first hint. As my job is to build websites while listening to music, I wasn't surprised to see Apache, Mysql and Spotify listening on ports they are supposed to listen on... But Avast points me in the right direction. Every opened ports (110, 143, 993, 995) are listed with "127.0.0.1.12XXX" (12110, 12143, 12993, 12995...).

I didn't succeed in viewing how Avast handles the real request on port 110 (@MarkPlotnick points me to use pfctl but using it is not that easy - any help welcomed).

I closed Avast and run lsof again, then telnet, then nmap... The false-opened ports were not there any more.

---

In such situation, I freaked out very quickly jumping on stackexchange to find a clue. I should have taken a breath, another coffee and think about what is in the path between my computer and my server. I would have tried other tests from other locations and find out that my server wasn't exposed to security breaches.

Conclusion : 1) Think 2) Think again 3) Test 4) Do