2

Somehow I keep breaking ssh on my ubuntu servers and I can't figure out why. I have a ddns set up as myserver.asuscomm.com and the ports are forwarded on my router. One of them works perfectly, but the other two show the ports as filtered when I connect from ddns. When I connect from the same network the ports are open.

Could this be some setting on my router that's wrong? Say the ssh port is 22100. netstat -tlpn shows that sshd is listening on this port and i have it to allow traffic from anywhere if i run sudo ufw status.

From the same network:

nmap -Pn -p 22100 192.168.50.x
Starting Nmap 7.80 ( https://nmap.org ) at 2021-10-01 10:13 CDT
Nmap scan report for 192.168.50.x
Host is up (0.00075s latency).

PORT      STATE SERVICE
22100/tcp open  optohost003

Nmap done: 1 IP address (1 host up) scanned in 0.13 seconds

From the ddns off network:

nmap -p 22100 myserver.asuscomm.com
Starting Nmap 7.80 ( https://nmap.org ) at 2021-10-01 10:14 CDT
Nmap scan report for myserver.asuscomm.com (x.y.z.123)
Host is up (0.025s latency).
rDNS record for x.y.z.123: c-x-y-z-123.abc.state.isp.net

PORT      STATE    SERVICE
22100/tcp filtered optohost003

Nmap done: 1 IP address (1 host up) scanned in 0.46 seconds

For this specific server it also says it's down when i do standard nmap so I have to add -Pn flag. The other server has the same issue but nmap can see it without -Pn.

Does anyone have ideas of how I can troubleshoot this? The ports are all forwarded on my Asus router and I can't figure out what other settings I could change. Should I try ssh on a different port?


Edit: output of netstat -nlp (port 22100 is my ssh)

tcp        0      0 127.0.0.1:32401         0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:22100           0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:32600         0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:41113         0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:445             0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:33060         0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:44935         0.0.0.0:*               LISTEN      -
tcp        0      0 192.168.50.7:27017      0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:27017         0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:139             0.0.0.0:*               LISTEN      -
tcp6       0      0 :::80                   :::*                    LISTEN      -
tcp6       0      0 :::32400                :::*                    LISTEN      -
tcp6       0      0 :::22100                :::*                    LISTEN      -
tcp6       0      0 :::445                  :::*                    LISTEN      -
tcp6       0      0 :::139                  :::*                    LISTEN      -
udp        0      0 127.0.0.53:53           0.0.0.0:*                           -
udp        0      0 192.168.50.7:68         0.0.0.0:*                           -
udp        0      0 172.17.255.255:137      0.0.0.0:*                           -
udp        0      0 172.17.0.1:137          0.0.0.0:*                           -
udp        0      0 192.168.50.255:137      0.0.0.0:*                           -
udp        0      0 192.168.50.7:137        0.0.0.0:*                           -
udp        0      0 0.0.0.0:137             0.0.0.0:*                           -
udp        0      0 172.17.255.255:138      0.0.0.0:*                           -
udp        0      0 172.17.0.1:138          0.0.0.0:*                           -
udp        0      0 192.168.50.255:138      0.0.0.0:*                           -
udp        0      0 192.168.50.7:138        0.0.0.0:*                           -
udp        0      0 0.0.0.0:138             0.0.0.0:*                           -
udp        0      0 127.0.0.1:33564         0.0.0.0:*                           -
udp        0      0 0.0.0.0:46034           0.0.0.0:*                           -
udp        0      0 0.0.0.0:5353            0.0.0.0:*                           -
udp        0      0 127.0.0.1:42818         0.0.0.0:*                           -
udp        0      0 0.0.0.0:51011           0.0.0.0:*                           -
udp        0      0 0.0.0.0:1901            0.0.0.0:*                           -
udp        0      0 192.168.50.7:38951      0.0.0.0:*                           -
udp        0      0 0.0.0.0:40101           0.0.0.0:*                           -
udp        0      0 0.0.0.0:32410           0.0.0.0:*                           -
udp        0      0 0.0.0.0:32412           0.0.0.0:*                           -
udp        0      0 0.0.0.0:32413           0.0.0.0:*                           -
udp        0      0 0.0.0.0:32414           0.0.0.0:*                           -
udp        0      0 192.168.50.7:57195      0.0.0.0:*                           -
udp        0      0 192.168.50.7:57254      0.0.0.0:*                           -
udp6       0      0 :::49275                :::*                                -
udp6       0      0 :::5353                 :::*                                -
raw6       0      0 :::58                   :::*                    7           -

output of sudo iptables -nvL

Chain INPUT (policy DROP 4793 packets, 1589K bytes)
 pkts bytes target     prot opt in     out     source               destination
 666K  472M ufw-before-logging-input  all  --  *      *       0.0.0.0/0            0.0.0.0/0
 666K  472M ufw-before-input  all  --  *      *       0.0.0.0/0            0.0.0.0/0
 262K   63M ufw-after-input  all  --  *      *       0.0.0.0/0            0.0.0.0/0
 155K   51M ufw-after-logging-input  all  --  *      *       0.0.0.0/0            0.0.0.0/0
 155K   51M ufw-reject-input  all  --  *      *       0.0.0.0/0            0.0.0.0/0
 155K   51M ufw-track-input  all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DOCKER-USER  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 DOCKER-ISOLATION-STAGE-1  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 DOCKER     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0
    0     0 ufw-before-logging-forward  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 ufw-before-forward  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 ufw-after-forward  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 ufw-after-logging-forward  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 ufw-reject-forward  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 ufw-track-forward  all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
 212K   16M ufw-before-logging-output  all  --  *      *       0.0.0.0/0            0.0.0.0/0
 212K   16M ufw-before-output  all  --  *      *       0.0.0.0/0            0.0.0.0/0
44581 3006K ufw-after-output  all  --  *      *       0.0.0.0/0            0.0.0.0/0
44581 3006K ufw-after-logging-output  all  --  *      *       0.0.0.0/0            0.0.0.0/0
44581 3006K ufw-reject-output  all  --  *      *       0.0.0.0/0            0.0.0.0/0
44581 3006K ufw-track-output  all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain DOCKER (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DOCKER-ISOLATION-STAGE-2  all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DROP       all  --  *      docker0  0.0.0.0/0            0.0.0.0/0
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain DOCKER-USER (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain ufw-after-forward (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain ufw-after-input (1 references)
 pkts bytes target     prot opt in     out     source               destination
  326 26112 ufw-skip-to-policy-input  udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:137
  730  180K ufw-skip-to-policy-input  udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:138
    2    88 ufw-skip-to-policy-input  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:139
    2    88 ufw-skip-to-policy-input  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:445
   27  8992 ufw-skip-to-policy-input  udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:67
    0     0 ufw-skip-to-policy-input  udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:68
 106K   11M ufw-skip-to-policy-input  all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type BROADCAST

Chain ufw-after-logging-forward (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 3/min burst 10 LOG flags 0 level 4 prefix "[UFW BLOCK] "

Chain ufw-after-logging-input (1 references)
 pkts bytes target     prot opt in     out     source               destination
 4337 1529K LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 3/min burst 10 LOG flags 0 level 4 prefix "[UFW BLOCK] "

Chain ufw-after-logging-output (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain ufw-after-output (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain ufw-before-forward (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0            icmptype 3
    0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0            icmptype 11
    0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0            icmptype 12
    0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0            icmptype 8
    0     0 ufw-user-forward  all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain ufw-before-input (1 references)
 pkts bytes target     prot opt in     out     source               destination
40237 2361K ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0
 308K  387M ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 ufw-logging-deny  all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate INVALID
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate INVALID
    0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0            icmptype 3
    0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0            icmptype 11
    0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0            icmptype 12
    0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0            icmptype 8
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp spt:67 dpt:68
 318K   82M ufw-not-local  all  --  *      *       0.0.0.0/0            0.0.0.0/0
 1987  325K ACCEPT     udp  --  *      *       0.0.0.0/0            224.0.0.251          udp dpt:5353
53945   19M ACCEPT     udp  --  *      *       0.0.0.0/0            239.255.255.250      udp dpt:1900
 262K   63M ufw-user-input  all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain ufw-before-logging-forward (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain ufw-before-logging-input (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain ufw-before-logging-output (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain ufw-before-output (1 references)
 pkts bytes target     prot opt in     out     source               destination
40237 2361K ACCEPT     all  --  *      lo      0.0.0.0/0            0.0.0.0/0
 128K   11M ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
44581 3006K ufw-user-output  all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain ufw-logging-allow (0 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 3/min burst 10 LOG flags 0 level 4 prefix "[UFW ALLOW] "

Chain ufw-logging-deny (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate INVALID limit: avg 3/min burst 10
    0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 3/min burst 10 LOG flags 0 level 4 prefix "[UFW BLOCK] "

Chain ufw-not-local (1 references)
 pkts bytes target     prot opt in     out     source               destination
 151K   51M RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL
60270   19M RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type MULTICAST
 107K   12M RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type BROADCAST
    0     0 ufw-logging-deny  all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 3/min burst 10
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain ufw-reject-forward (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain ufw-reject-input (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain ufw-reject-output (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain ufw-skip-to-policy-forward (0 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain ufw-skip-to-policy-input (7 references)
 pkts bytes target     prot opt in     out     source               destination
 107K   12M DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain ufw-skip-to-policy-output (0 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain ufw-track-forward (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain ufw-track-input (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain ufw-track-output (1 references)
 pkts bytes target     prot opt in     out     source               destination
  173 10348 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate NEW
44407 2995K ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate NEW

Chain ufw-user-forward (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain ufw-user-input (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     tcp  --  *      *       73.6.251.121         0.0.0.0/0            tcp dpt:27017
    0     0 ACCEPT     udp  --  *      *       73.6.251.121         0.0.0.0/0            udp dpt:27017
   53  2812 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:22001
    6   360 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:22000
    3   132 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:21
    6   344 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:22100

Chain ufw-user-limit (0 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 3/min burst 5 LOG flags 0 level 4 prefix "[UFW LIMIT BLOCK] "
    0     0 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable

Chain ufw-user-limit-accept (0 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain ufw-user-logging-forward (0 references)
 pkts bytes target     prot opt in     out     source               destination

Chain ufw-user-logging-input (0 references)
 pkts bytes target     prot opt in     out     source               destination

Chain ufw-user-logging-output (0 references)
 pkts bytes target     prot opt in     out     source               destination

Chain ufw-user-output (1 references)
 pkts bytes target     prot opt in     out     source               destination

iptables -t nat -nvL:

Chain PREROUTING (policy ACCEPT 35687 packets, 8678K bytes)
 pkts bytes target     prot opt in     out     source               destination
 4099 1364K DOCKER     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 6440 packets, 1931K bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 3131 packets, 314K bytes)
 pkts bytes target     prot opt in     out     source               destination
    1   278 DOCKER     all  --  *      *       0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 3131 packets, 314K bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 MASQUERADE  all  --  *      !docker0  172.17.0.0/16        0.0.0.0/0

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 RETURN     all  --  docker0 *       0.0.0.0/0            0.0.0.0/0

iptables -t mangle -nvL:

Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Another edit: To add to this, I finally checked var/log/syslog and noticed something interesting: ufw is blocking the connection from my router to the server.

Oct  1 16:36:00 user kernel: [   71.994434] [UFW BLOCK] IN=eth0 OUT= MAC=mac_address_here SRC=192.168.50.1 DST=192.168.50.7 LEN=386 TOS=0x00 PREC=0x00 TTL=64 ID=49481 PROTO=UDP SPT=1900 DPT=56241 LEN=366 

I set up ufw to allow traffic to and from 192.168.50.1 but no dice. hmm...

3

Let's assume that myserver.asuscomm.com really points to your server's router, and that your router is correctly forwarding ports to the server.

A service can be listening and active on the loopback, or from a LAN address, but not available elsewhere. This seems to demonstrate what you are seeing.

Therefore you need to look at,

  • what's listening on the ports
  • what interfaces/addresses these services are listening on
  • what firewall rules you have implemented
  • what application level connection rules you have implemented

The first two of these can be seen with a command such as netstat -nlp. You provided these details in your question:

Proto Recv-Q Send-Q Local Address  Foreign Address  State   PID/Program name
tcp        0      0 0.0.0.0:22100  0.0.0.0:*        LISTEN  -
tcp6       0      0 :::22100       :::*             LISTEN  -

These confirm that your service on port 22100 is listening on all interfaces (0.0.0.0 and :::) for any inbound traffic.

Firewall rules implemented with iptables can be checked with commands such as these (if you're using nftables I can't help you directly):

iptables -nvL
iptables -t nat -nvL
iptables -t mangle -nvL

And looking at the output you've added to your question I can see that there is an ACCEPT rule in the UFW firewall ruleset that permits inbound TCP traffic to port 22100:

Chain ufw-user-input (1 references)
 pkts bytes target     prot opt in     out     source               destination
  […]
    6   344 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:22100

Application level connection rules are usually defined in /etc/hosts.allow and/or /etc/hosts.deny, but can also be found in the application-specific configuration files (/etc/ssh/sshd_config, for example).

I don't see anything wrong with the server setup here, so I would double check that,

  • you are connecting from outside to the correct IP address with the correct port (some of your examples show port 1234 and others reference port 22100)
  • your ISP permits inbound TCP/IP traffic on this port
  • your router's port forwarding really is set up correctly

As an aside, you may have a terminology problem. "when I connect from ddns" doesn't really make sense. You can't connect "from" DDNS as it's just a service that translates domain names to (dynamic) IP addresses. I think you're saying "when I connect from outside my network to the IP address resolved through DDNS".

11
  • 1
    thank you for the thorough response (and the aside!). i have edited the question to include outputs of the commands you suggested. I will check /etc/hosts.allow and etc/hosts.deny as well Oct 1 at 16:36
  • done. that's what i'm really confused about. all i did was change the ports in /etc/ssh/sshd_config but something is filtering the packets for traffic coming from outside of the network Oct 1 at 17:42
  • i did and i realized the one that was working had ufw disabled so i disabled it for the other two. one of them is connecting but the other is getting an ERROR: async io completed with error: 10060, io:0000027C601D3FB0 when trying to ssh from off network Oct 1 at 17:52
  • ssh -vvv -p 22001 user@myserver.asuscomm.com (I changed the port to line up with my other servers). it's just so weird b/c it works fine if i connect to another server and then do server-to-server ssh within the network Oct 1 at 19:50
  • 22001 or 22100?
    – roaima
    Oct 1 at 19:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.