Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

This is a Drupal website running Ubuntu 12.04 LTS on Linode. Someone recently changed our iptables rules and a number of problems started.

These include:

  1. Our Mollom spam protection is not working on the Drupal site because the server cannot reach the external service. (more details below)

  2. Drupal reports, "Your system or network configuration does not allow Drupal to access web pages, resulting in reduced functionality." (more details below)

  3. the server cannot ping the Internet

    ping google.com
    ping: unknown host google.com

  4. Other network services fail. For example:

    $ sudo ntpdate ntp.ubuntu.com Exiting, name server cannot be used: Temporary failure in name resolution (-3) 1 Jun 16:42:34 ntpdate[7420]: name server cannot be used: Temporary failure in name resolution (-3)
    apt-get update Err http://us.archive.ubuntu.com precise Release.gpg
    Temporary failure resolving 'us.archive.ubuntu.com' Err http://security.ubuntu.com precise-security Release.gpg
    Temporary failure resolving 'security.ubuntu.com'

  5. The server is running very poorly. Could errors like the following one be related to inability to access the Internet due to iptables rules?

    Out of memory: Kill process 2300 (mysqld) score 129 or sacrifice child Killed process 2300 (mysqld) total-vm:354780kB, anon-rss:53180kB, file-rss:0kB Out of memory: Kill process 5937 (mysqld) score 60 or sacrifice child Killed process 5937 (mysqld) total-vm:344040kB, anon-rss:78988kB, file-rss:0kB

Here are the Drupal errors in more detail. Drupal configuration was not changed -- only iptables was changed afaik:

1. HTTP request status Fails Your system or network configuration does not allow Drupal to access web pages, resulting in reduced functionality. This could be due to your webserver configuration or PHP settings, and should be resolved in order to download information about available updates, fetch aggregator feeds, sign in via OpenID, or use other network-dependent services. If you are certain that Drupal can access web pages but you are still seeing this message, you may add $conf['drupal_http_request_fails'] = FALSE; to the bottom of your settings.php file.

2. Mollom API keys Service error The Mollom API keys could not be verified. Please try again later. This issue prevents members from registering for our site or posting comments.

Here are the current (problematic) iptables rules. (BTW, port 2222 was originally the SSH port. Now someone changed SSH back to port 22, but apparently left port 2222 open in iptables, but that's a separate issue):

# iptables -nL
Chain INPUT (policy DROP)
target     prot opt source               destination         
fail2ban-apache-overflows  tcp  --  0.0.0.0/0            0.0.0.0/0            multiport dports 80,443
fail2ban-apache  tcp  --  0.0.0.0/0            0.0.0.0/0            multiport dports 80,443
DROP       all  --  69.30.238.162        0.0.0.0/0           
DROP       all  --  121.205.199.140      0.0.0.0/0           
DROP       all  --  137.117.129.250      0.0.0.0/0           
DROP       all  --  117.26.252.26        0.0.0.0/0           
DROP       all  --  112.111.165.71       0.0.0.0/0           
DROP       all  --  175.42.95.145        0.0.0.0/0           
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:80 state NEW,ESTABLISHED
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:2222 state NEW,ESTABLISHED
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:443 state NEW,ESTABLISHED
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:22
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:2222

Chain FORWARD (policy DROP)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp spt:80 state ESTABLISHED
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp spt:2222 state ESTABLISHED
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp spt:443 state ESTABLISHED
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp spt:22
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp spt:2222

Chain fail2ban-apache (1 references)
target     prot opt source               destination         
RETURN     all  --  0.0.0.0/0            0.0.0.0/0           

Chain fail2ban-apache-overflows (1 references)
target     prot opt source               destination         
RETURN     all  --  0.0.0.0/0            0.0.0.0/0  

My question: since I know nothing about fail2ban configuration, what are the minimal changes I can make that will restore basic functionality (especially for outgoing services as those appear to be the main problem)?

UPDATE: based on answer from Michael Kjörling, here are the updated iptables rules. Is this correct?

# cat iptables.modified.rules 
# Generated by iptables-save v1.4.12 on Sat Jun  1 21:26:17 2013
*security
:INPUT ACCEPT [60016:6837978]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [60450:61970107]
COMMIT
# Completed on Sat Jun  1 21:26:17 2013
# Generated by iptables-save v1.4.12 on Sat Jun  1 21:26:17 2013
*raw
:PREROUTING ACCEPT [72450:7582406]
:OUTPUT ACCEPT [60450:61970107]
COMMIT
# Completed on Sat Jun  1 21:26:17 2013
# Generated by iptables-save v1.4.12 on Sat Jun  1 21:26:17 2013
*nat
:PREROUTING ACCEPT [7899:447793]
:INPUT ACCEPT [7787:439884]
:OUTPUT ACCEPT [5818:4357219]
:POSTROUTING ACCEPT [5818:4357219]
COMMIT
# Completed on Sat Jun  1 21:26:17 2013
# Generated by iptables-save v1.4.12 on Sat Jun  1 21:26:17 2013
*mangle
:PREROUTING ACCEPT [72450:7582406]
:INPUT ACCEPT [72450:7582406]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [60450:61970107]
:POSTROUTING ACCEPT [60450:61970107]
COMMIT
# Completed on Sat Jun  1 21:26:17 2013
# Generated by iptables-save v1.4.12 on Sat Jun  1 21:26:17 2013
*filter
:INPUT DROP [44:2387]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [78:41503]
:fail2ban-apache - [0:0]
:fail2ban-apache-overflows - [0:0]
-A INPUT -s 69.30.238.162/32 -j DROP
-A INPUT -s 121.205.199.140/32 -j DROP
-A INPUT -s 137.117.129.250/32 -j DROP
-A INPUT -s 117.26.252.26/32 -j DROP
-A INPUT -s 112.111.165.71/32 -j DROP
-A INPUT -s 175.42.95.145/32 -j DROP
-A INPUT -p tcp -m multiport --dports 80,443 -j fail2ban-apache-overflows
-A INPUT -p tcp -m multiport --dports 80,443 -j fail2ban-apache
-A INPUT -i eth0 -p tcp -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 443 -m state --state NEW,ESTABLISHED -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 80 -m state --state NEW,ESTABLISHED -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -p udp -m state --state RELATED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A fail2ban-apache -j RETURN
-A fail2ban-apache-overflows -j RETURN
COMMIT
# Completed on Sat Jun  1 21:26:17 2013
share|improve this question
    
Aside from this question, I think you should be concerned about "someone recently did X". You need to establish proper procedures for tracking changes in production environments, such that you can go to the person who authorized or performed a change and discuss it with them. You have two "someone did something at some point" right in this question; who knows what other surprises might be lurking for you in your production environment? –  Michael Kjörling Jun 1 '13 at 21:43
    
The first "someone" was our part time sysadmin who has made so many mistakes that I hope to not use his services again. The second "someone" is the replacement (so far) for the first. Obviously, this is not a good situation. What else to do when you cannot afford to hire the right people? I guess you come here and ask questions ;-) –  MountainX Jun 1 '13 at 21:49

1 Answer 1

up vote 3 down vote accepted

That the OOM killer is kicking in may well be related to your problems, and it most definitely doesn't help you. Increase swap or install more RAM. If nothing else, you certainly don't want your server going around killing processes; by the time the kernel OOM killer starts looking for victims, it's bad.

That said, I do notice that you e.g. don't allow in ICMP, or DNS over UDP (or TCP). A well-behaved host on the Internet needs to accept ICMP packets, and if you are running a DNS server you will need to open up communication over TCP and UDP to port 53 for it to work.

A minimal change that should bring you more in line with a reasonable configuration without affecting what services are available to external parties might be to simply

iptables -A INPUT -p tcp -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -p udp -m state --state RELATED -j ACCEPT
iptables -A INPUT -p icmp -j ACCEPT

(Untested, so the command line syntax may be slightly off, but that's the gist of it.)

Normally, for performance reasons such rules would go at or near the top of the iptables chain in question. These will allow in any responses to anything outbound the server has initiated over TCP or UDP, as well as ICMP messages. Note that ICMP is crucial for a well-behaved host on the Internet; if you want to block e.g. ping, you can use a much more specific rule to block ICMP echo requests and nothing else, but when you have something as well-known as port 80 wide open that doesn't buy you much in practice and it does make troubleshooting more complicated.

The output chain rules are also all redundant as they do nothing that the chain policy doesn't do. Just iptables -F OUTPUT to remove them; they complicate the configuration and cost a tiny bit of performance for every outgoing packet without providing any benefit whatsoever. They also specify for the most part privileged source ports, which are almost guaranteed to not match what you want.

share|improve this answer
    
Thank you. I would like to paste my new saved iptables rules to check whether they are correct. I can't post that here in a comment. I'll add it to my question. –  MountainX Jun 1 '13 at 21:34
    
@MountainX The best way to determine if the new set of rules is "correct" is to see whether they do what you need. You are in a much better position to judge that than some random person on the other side of the Internet. –  Michael Kjörling Jun 1 '13 at 21:39
    
It is my understanding that changes to the iptables rules take effect immediately. I made the changes and saved the new rules and posted them in my question. However, after making the changes, the problems are not resolved. For example, I still cannot ping out to google.com or run apt-get update. –  MountainX Jun 1 '13 at 21:46
    
@MountainX Can you ping -n to a known-good IP address? Also, some of your rules specify eth0 explicitly; is that correct? (How would I know?) –  Michael Kjörling Jun 1 '13 at 21:47
    
eth0 is correct –  MountainX Jun 1 '13 at 21:50

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.