I have a flannel-based k8s system with a container at
I also have an interface
ens224:1 with the IP address
With older kernels (3.x) I'm able to force packets destined to
10.5.2.11 to be routed to the container using a
fwmark and an
ip rule, and matching
ip route entries. This is useful for dealing with ingress locally in k8s instead of the usual "ingress is someone else's problem" that is typical with k8s. (I want full control: I'm running a custom router on a raw socket in that container).
This has stopped working in 4.x kernels I've tried. The packets (e.g. ping) just dead end. I've verified I have
rp_filter set to 0 and martian packet logging enabled (and no such log messages).
If I ping from a remote machine, the packets to
10.5.2.11 come in to
ens224 but go no where. If I ping
10.5.2.11 from the local machine packets get routed correctly.
The following is the config that works in 3.x kernels:
-A PREROUTING -d 10.5.2.11/32 -j MARK --set-xmark 0x1/0x1
-A OUTPUT -d 10.5.2.11/32 -j MARK --set-xmark 0x1/0x1
49: from iif l0 lookup 245
50: from all fwmark 0x1/0x1 lookup 245
100 from all lookup 246
ip route show table 245
10.5.2.11 via 10.233.64.18 dev cni0
Table 246 shows the local route that would be normally in table
local, which I've deleted from table
local. I moved it to table 246 so that packets are not routed to the Linux socket listener tables. But at the same time I want the interface to respond to ARPs.
ip route show table 246
local 10.5.2.11 dev ens224 proto kernel scope host src 10.5.2.11
ip route delete 10.5.2.11 table local
And for those not familiar with CNI, it's just a bridge. In the non-working condition, I see no ICMP packets on
cni0 on 4.x, but I do see them get routed to
cni0 in 3.x
port cni0 vlan 1 ids PVID Egress Untaged
So, why does this arrangement work on Linux version 3.x but not 4.x? Or perhaps there's something else causing the iCMP (or TCP, or UDP..) packets to get routed nowhere?