Linux Webserver not "Accepting" Connections

Question

First off, thanks for any help anyone can provide, I greatly appreciate it!

I have a basic network I have set up for testing. It has pfSense acting as a Gateway/DHCP/DNS host and two Linux machines behind it. One of the Linux machines is hosting an Apache web server, where I am having issues getting the webpage to load from the other Linux machine.

I thought this had to do with httpd binding to the loopback IP Address but I seem to have resolved that and the issue still exists. Looking at the last nmap, it looks like the webserver is not opening port 80 or 443 for external access.

Here is the netstat output on the webserver:

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:3306            0.0.0.0:*               LISTEN      1502/mariadbd       
tcp        0      0 127.0.0.1:44321         0.0.0.0:*               LISTEN      1570/pmcd           
tcp        0      0 127.0.0.1:60999         0.0.0.0:*               LISTEN      809/glance-apiuWSGI 
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      760/systemd-resolve 
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1199/sshd: /usr/sbi 
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      1247/httpd          
tcp        0      0 0.0.0.0:8775            0.0.0.0:*               LISTEN      811/nova-api-metauW 
tcp        0      0 0.0.0.0:25672           0.0.0.0:*               LISTEN      1190/beam.smp       
tcp        0      0 0.0.0.0:9696            0.0.0.0:*               LISTEN      820/ml2_conf.ini    
tcp        0      0 127.0.0.54:53           0.0.0.0:*               LISTEN      760/systemd-resolve 
tcp        0      0 127.0.0.1:6640          0.0.0.0:*               LISTEN      1013/ovsdb-server   
tcp        0      0 0.0.0.0:6642            0.0.0.0:*               LISTEN      937/ovsdb-server    
tcp        0      0 0.0.0.0:6641            0.0.0.0:*               LISTEN      909/ovsdb-server    
tcp        0      0 127.0.0.1:39711         0.0.0.0:*               LISTEN      811/nova-api-metauW 
tcp        0      0 20.20.20.11:2379        0.0.0.0:*               LISTEN      1169/etcd           
tcp        0      0 127.0.0.1:4330          0.0.0.0:*               LISTEN      2234/pmlogger       
tcp        0      0 0.0.0.0:5355            0.0.0.0:*               LISTEN      760/systemd-resolve 
tcp        0      0 0.0.0.0:6080            0.0.0.0:*               LISTEN      815/python3.11      
tcp6       0      0 :::2380                 :::*                    LISTEN      1169/etcd           
tcp6       0      0 ::1:4330                :::*                    LISTEN      2234/pmlogger       
tcp6       0      0 :::22                   :::*                    LISTEN      1199/sshd: /usr/sbi 
tcp6       0      0 :::443                  :::*                    LISTEN      1247/httpd          
tcp6       0      0 :::9090                 :::*                    LISTEN      1/systemd           
tcp6       0      0 :::4369                 :::*                    LISTEN      1/systemd           
tcp6       0      0 ::1:44321               :::*                    LISTEN      1570/pmcd           
tcp6       0      0 :::5355                 :::*                    LISTEN      760/systemd-resolve 
tcp6       0      0 :::5672                 :::*                    LISTEN      1190/beam.smp  

Here are my IP Tables on the web server:

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED
ACCEPT     icmp --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     tcp  --  anywhere             anywhere             state NEW tcp dpt:ssh
REJECT     all  --  anywhere             anywhere             reject-with icmp-host-prohibited
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:http
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:https

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
REJECT     all  --  anywhere             anywhere             reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination      type here

Here is an nmap on the web server itself:

Starting Nmap 7.93 ( https://nmap.org ) at 2023-08-09 19:18 CDT
Nmap scan report for opendevhost.testing.prox (20.20.20.11)
Host is up (0.0000080s latency).
Not shown: 995 closed tcp ports (reset)
PORT     STATE SERVICE
22/tcp   open  ssh
80/tcp   open  http
443/tcp  open  https
3306/tcp open  mysql
9090/tcp open  zeus-admin

Nmap done: 1 IP address (1 host up) scanned in 0.09 seconds

Here is an nmap on the webserver from the other Linux box on the LAN:

Starting Nmap 7.93 ( https://nmap.org ) at 2023-08-09 19:17 CDT
Nmap scan report for opendevhost.testing.prox (20.20.20.11)
Host is up (0.00041s latency).
Not shown: 988 filtered tcp ports (no-response), 10 filtered tcp ports (admin-prohibited)
PORT     STATE SERVICE
22/tcp   open  ssh
9090/tcp open  zeus-admin
MAC Address: 46:0E:A2:7D:12:1A (Unknown)

Nmap done: 1 IP address (1 host up) scanned in 5.20 seconds


I added http and https services through the firewall by logging into the server via the web console.

Answer 1

You added enabling rules after a generic REJECT rule that will always match: your additional rules will never be traversed and are thus useless. Terminal rules are always immediate: once REJECT is evaluated, there will be no further rule evaluated in this hook (INPUT).

One way to fix this is to remove (wherever it is) and put back (at the end) the REJECT rule. While at it, precede the REJECT rule with a rule dropping INVALID states, to avoid REJECT to reject established flows in rare cases involving delayed packets (as documented in REJECT's warning in newer versions).

iptables -D INPUT -j REJECT
iptables -A INPUT -m conntrack --ctstate INVALID -j DROP
iptables -A INPUT -j REJECT

It's better to rebuild the ruleset from a know good source, either with a script, a pair of iptables-save to save it in a file / iptables-restore to restore it from this file or a firewall framework such as firewalld or UFW.