Why do iptables drop / log even if the appropriate packet should be accepted?

Question

I'm setting up a server running Debian Jessie with some applications like iptables firewall, fail2ban, openvpn, apache, ...

The iptables firewall is configured in the way, that it logs every packet that is dropped. A small excerpt from the iptables configuration:

...
-A INPUT -m comment --comment "003 accept related established rules IPv4" -m state --state RELATED,ESTABLISHED -j ACCEPT
...
-A INPUT -p tcp -m multiport --dports 1194 -m comment --comment "303 allow incoming OpenVPN" -m state --state NEW -j ACCEPT
...
-A INPUT -m comment --comment "900 IPv4 log dropped input chain" -j LOG --log-prefix "[IPTABLES INPUT IPv4] DROP " --log-level 6
-A INPUT -m comment --comment "910 IPv4 deny all other input requests" -j DROP

OpenVPN (which uses port 1194) works well. I can connect, use the connection and work for hours, up to the point when I need to transfer larger amount of data. At this point some lines like the following appear in the log:

Mar 02 20:39:27 rs3 kernel: [IPTABLES INPUT IPv4] DROP IN=eth0 OUT= MAC=ae:12:7b:9b:5d:e4:00:15:c7:c9:45:80:08:00 SRC=<MyIPAddress> DST=<ServerIPAddress> LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=20713 DF PROTO=TCP SPT=61941 DPT=1194 WINDOW=3040 RES=0x00 ACK URGP=0
Mar 02 20:39:27 rs3 kernel: [IPTABLES INPUT IPv4] DROP IN=eth0 OUT= MAC=ae:12:7b:9b:5d:e4:00:15:c7:c9:45:80:08:00 SRC=<MyIPAddress> DST=<ServerIPAddress> LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=20718 DF PROTO=TCP SPT=61941 DPT=1194 WINDOW=3040 RES=0x00 ACK URGP=0 
Mar 02 20:39:27 rs3 kernel: [IPTABLES INPUT IPv4] DROP IN=eth0 OUT= MAC=ae:12:7b:9b:5d:e4:00:15:c7:c9:45:80:08:00 SRC=<MyIPAddress> DST=<ServerIPAddress> LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=20719 DF PROTO=TCP SPT=61941 DPT=1194 WINDOW=3040 RES=0x00 ACK URGP=0 

[These logs are picked up from fail2ban and the server closes the connection to my local computer.]

Q: Why are those packets dropped (and logged)? That means: Why are those packets not processed by the 'RELATED,ESTABLISHED' rule?

What I did so far: read documentation, thought about the problem ;-), checked the rules again and again, googled the Internet - but without any result.

Edit: maybe this is important: the DROP list that was created by fail2ban has about 500 entries.

Edit: I want to know the reason (this is why I'm using 'Why' in my question). I'm not interested in workarounds.

Edit: This behavior is not limited to OpenVPN, using other protocols (e.g. ssh) has the same problem.

Edit: To give you an impression, have a look at the wireshark screenshot: the highlighted packet was droped / logged - all other packets from the screenshot not.

Here is the appropriate log:

Mar 03 10:16:23 rs3 kernel: [IPTABLES INPUT IPv4] DROP IN=eth0 OUT= MAC=ae:12:7b:9b:5d:e4:00:15:c7:c9:45:80:08:00 SRC=<MyIPAddress>.117 DST=<ServerIPAddress>.116 LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=11550 DF PROTO=TCP SPT=63526 DPT=1194 WINDOW=32038 RES=0x00 ACK URGP=0 

I used the IP ID (11550) to find the packet in wireshark. And this is the only IP packet with this ID.

Edit: The server IP is a fixed IP. My dynamically assigned IP address was not changed during the the test. Connections are initiated by the LocalComputer towards the Server. The setup is as follows:

=================      ===================      ===============
| LocalComputer | ---- | NAT Router .117 | ---- | Server .116 |
=================      ===================      ===============
   Private IP            Dynam. Assigned            Fixed IP
                       but fixed during test

Answer 1

Because you've got those iptables log entries, and they haven't matched your RELATED,ESTABLISHED rules, the connection tracker is reporting seeing them as the start of new sessions unrelated to anything earlier. Typically (but not always) this can happen when the session has spent longer idle than the connection tracker timeout. Other NAT devices between your endpoints might have timed out and generated new port numbers, or your other endpoint might have had its IP address changed (possible on some ISPs' DHCP-based networks) so it's not necessarily the endpoint connection tracker that is at fault.

There are three options that spring to mind,

1. Enable ping keepalive within the OpenVPN configuration. As a minimum, try ping 10. If you prefer, and depending on how your client/server is set up you may prefer keepalive 10 60. See the man page for really gory details. (This won't solve the problem if the IP address/port pair for an endpoint changes.)

2. Disable the fail2ban configuration that matches OpenVPN traffic on 1194/tcp.

3. Ensure that you avoid printing a log message from iptables for OpenVPN traffic on 1194/tcp, so fail2ban has nothing on which to trigger.

Answer 2

The connection tracker does not only track, if packets are from the same connection, but also checks for some timings (like late ACK or SEQ too late).

It is possible to switch on connection tracker logging with:

echo 255 >/proc/sys/net/netfilter/nf_conntrack_log_invalid

This showed me, that all dropped and logged packages are of the type:

nf_ct_tcp: ACK is under the lower bound (possible overly delayed ACK) ...

There is a discussion where I found these hints.

I am currently not aware why this happens, but this might be another question...