Ever growing iptables

Question

My iptables keeps growing...

When I do sudo iptables -L | wc -l, it shows a number, if I do it again later that number goes up.

It seems to increase by 1 every 2 seconds.

---

I tried to reboot, not changing anything...

---

I'm on Ubuntu 16.04.6

EDIT #1

I did like @Artem mentioned :

sudo iptables-save > /tmp/ipt.dump1
# Wait a few seconds
sudo iptables-save > /tmp/ipt.dump2
diff -u /tmp/ipt.dump1 /tmp/ipt.dump2

And I got this result :

--- /tmp/ipt.dump1      2020-07-20 17:39:27.443308154 +0900
+++ /tmp/ipt.dump2      2020-07-20 17:39:40.831173660 +0900
@@ -1,9 +1,9 @@
-# Generated by iptables-save v1.6.0 on Mon Jul 20 17:39:27 2020
+# Generated by iptables-save v1.6.0 on Mon Jul 20 17:39:40 2020
 *nat
-:PREROUTING ACCEPT [545:78025]
-:INPUT ACCEPT [545:78025]
-:OUTPUT ACCEPT [2686:163879]
-:POSTROUTING ACCEPT [2686:163879]
+:PREROUTING ACCEPT [547:78264]
+:INPUT ACCEPT [547:78264]
+:OUTPUT ACCEPT [2726:166287]
+:POSTROUTING ACCEPT [2726:166287]
 :DOCKER - [0:0]
 -A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
 -A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
@@ -12,12 +12,12 @@
 -A POSTROUTING -s 172.17.0.2/32 -d 172.17.0.2/32 -p tcp -m tcp --dport 8501 -j MASQUERADE
 -A DOCKER -i docker0 -j RETURN
 COMMIT
-# Completed on Mon Jul 20 17:39:27 2020
-# Generated by iptables-save v1.6.0 on Mon Jul 20 17:39:27 2020
+# Completed on Mon Jul 20 17:39:40 2020
+# Generated by iptables-save v1.6.0 on Mon Jul 20 17:39:40 2020
 *filter
-:INPUT ACCEPT [11:1274]
+:INPUT ACCEPT [38:5571]
 :FORWARD DROP [0:0]
-:OUTPUT ACCEPT [7:464]
+:OUTPUT ACCEPT [9:720]
 :DOCKER - [0:0]
 :DOCKER-ISOLATION-STAGE-1 - [0:0]
 :DOCKER-ISOLATION-STAGE-2 - [0:0]
@@ -829,6 +829,14 @@
 -A INPUT -s 127.0.0.1/32 -j ACCEPT
 -A INPUT -s 127.0.0.1/32 -j ACCEPT
 -A INPUT -s 127.0.0.1/32 -j ACCEPT
+-A INPUT -s 127.0.0.1/32 -j ACCEPT
+-A INPUT -s 127.0.0.1/32 -j ACCEPT
+-A INPUT -s 127.0.0.1/32 -j ACCEPT
+-A INPUT -s 127.0.0.1/32 -j ACCEPT
+-A INPUT -s 127.0.0.1/32 -j ACCEPT
+-A INPUT -s 127.0.0.1/32 -j ACCEPT
+-A INPUT -s 127.0.0.1/32 -j ACCEPT
+-A INPUT -s 127.0.0.1/32 -j ACCEPT
 -A FORWARD -j DOCKER-USER
 -A FORWARD -j DOCKER-ISOLATION-STAGE-1
 -A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
@@ -841,4 +849,4 @@
 -A DOCKER-ISOLATION-STAGE-2 -j RETURN
 -A DOCKER-USER -j RETURN
 COMMIT
-# Completed on Mon Jul 20 17:39:27 2020
+# Completed on Mon Jul 20 17:39:40 2020

EDIT #2

I followed this to track processes :

git clone https://github.com/brendangregg/perf-tools.git
cd perf-tools
sudo ./execsnoop

I then filtered to see only things related to iptables :

sudo ./execsnoop | grep iptables

And every 1.5 seconds I have 2 new processes :

  8596   8595 iptables -L INPUT
  8599   8534 iptables -I INPUT -s 127.0.0.1 -j ACCEPT
  8705   8704 iptables -L INPUT
  8708   8643 iptables -I INPUT -s 127.0.0.1 -j ACCEPT
  8814   8813 iptables -L INPUT
  8817   8752 iptables -I INPUT -s 127.0.0.1 -j ACCEPT
  8923   8922 iptables -L INPUT
  8926   8861 iptables -I INPUT -s 127.0.0.1 -j ACCEPT
  9033   9032 iptables -L INPUT
  9036   8971 iptables -I INPUT -s 127.0.0.1 -j ACCEPT
  9142   9141 iptables -L INPUT
  9145   9080 iptables -I INPUT -s 127.0.0.1 -j ACCEPT
  9251   9250 iptables -L INPUT
  9254   9189 iptables -I INPUT -s 127.0.0.1 -j ACCEPT
  9360   9359 iptables -L INPUT
  9363   9298 iptables -I INPUT -s 127.0.0.1 -j ACCEPT
  9469   9468 iptables -L INPUT
  9472   9407 iptables -I INPUT -s 127.0.0.1 -j ACCEPT
  9578   9577 iptables -L INPUT
  9581   9516 iptables -I INPUT -s 127.0.0.1 -j ACCEPT
  9687   9686 iptables -L INPUT
  9690   9625 iptables -I INPUT -s 127.0.0.1 -j ACCEPT

But it doesn't help me finding the root cause...

Answer 1

Please do the following:

sudo iptables-save > /tmp/ipt.dump1

a few seconds later

sudo iptables-save > /tmp/ipt.dump2

Then please post diff -u /tmp/ipt.dump1 /tmp/ipt.dump2

At least you'll know what you're dealing with.

---

As a last resort, I'd do the following:

sudo mv /sbin/iptables /sbin/iptables.real

and then create a bash script instead, e.g.

#! /bin/bash
echo "`date`: I was called by $PPID `readlink /proc/$PPID/exe` " >> /tmp/iptables.log
/sbin/iptables.real "$@"

This will let you find out what process is calling iptables incessantly.

Answer 2

This awk script might help you find who's calling iptables. It pieces together a process tree from execsnoop's output (pid ppid command ...). The parents are indented and appear after the children, sorry; with more time, I could get it to look like pstree, but this may do.

Run execsnoop for a few seconds with its output diverted to a file, then run this with input from that file.

awk -v search=iptables '
    $1 ~ /^[0-9]+$/ {
        p = $1
        ppid[p] = $2
        # command is in $3..$NF, so we'll
        # remove the first 2 fields, then store $0
        for (i = 3; i <= NF; i++) {
            $(i-2) = $i
        }
        NF -= 2
        cmd[p] = $0
    }
    END {
        for (p in cmd) {
            if (cmd[p] ~ search) {
                prefix = ""
                do {
                    print prefix, p, cmd[p]
                    p = ppid[p]
                    prefix = prefix " "
                } while (p)
            }
        }
    }'

Answer 3

Turned out, it was a virus...

I opened another stack exchange post on what to do next.

---

I'm accepting this answer because it's the answer of my specific case, but be sure to check comments and other answers, which provide really useful tools to track the potential cause of this.