0

I have a flannel-based k8s system with a container at 10.233.64.18.

I also have an interface ens224:1 with the IP address 10.5.2.11

With older kernels (3.x) I'm able to force packets destined to 10.5.2.11 to be routed to the container using a fwmark and an ip rule, and matching ip route entries. This is useful for dealing with ingress locally in k8s instead of the usual "ingress is someone else's problem" that is typical with k8s. (I want full control: I'm running a custom router on a raw socket in that container).

This has stopped working in 4.x kernels I've tried. The packets (e.g. ping) just dead end. I've verified I have rp_filter set to 0 and martian packet logging enabled (and no such log messages).

If I ping from a remote machine, the packets to 10.5.2.11 come in to ens224 but go no where. If I ping 10.5.2.11 from the local machine packets get routed correctly.

The following is the config that works in 3.x kernels:

IP Tables:

-A PREROUTING -d 10.5.2.11/32 -j MARK --set-xmark 0x1/0x1
-A OUTPUT -d 10.5.2.11/32 -j MARK --set-xmark 0x1/0x1

IP Rules:

49: from iif l0 lookup 245
50: from all fwmark 0x1/0x1 lookup 245
100 from all lookup 246

ip route show table 245

10.5.2.11 via 10.233.64.18 dev cni0

Table 246 shows the local route that would be normally in table local, which I've deleted from table local. I moved it to table 246 so that packets are not routed to the Linux socket listener tables. But at the same time I want the interface to respond to ARPs.

ip route show table 246

local 10.5.2.11 dev ens224 proto kernel scope host src 10.5.2.11
ip route delete 10.5.2.11 table local

And for those not familiar with CNI, it's just a bridge. In the non-working condition, I see no ICMP packets on cni0 on 4.x, but I do see them get routed to cni0 in 3.x

bridge vlan
port cni0 vlan 1 ids PVID Egress Untaged
[...]

So, why does this arrangement work on Linux version 3.x but not 4.x? Or perhaps there's something else causing the iCMP (or TCP, or UDP..) packets to get routed nowhere?

  • I'm not familiar with flannel, and I don't understand your goal completely, but (1) ens224:1 is not an "interface", it's just the old way to add multiple IP addresses to the interface ens224, (2) the local table etc. is a bit involved (I'd have to look this up), and messing with it will likely break things, as this probably changed from 3.x to 4.x, (3) if all you want is an interface in a container which acts like it was a part of ens224 with an IP address of 10.5.2.11/32, just use a macvlan interface in your container with ens224 as master. No routing needed. – dirkt Dec 18 '18 at 7:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.