2

On a system are VRFs, multiple routing tables and rules defined. A local service (SSH) is not reachable via one interface (WAN interface that is in a VRF) while it is reachable from other interfaces. Packet filter is disabled.

I would like to debug where things are getting stuck exactly.

How exactly is a route decided is vrf and ip rules are defined at the same time?

As an example, in absence of VRFs:

# ip rule show
101:    from all lookup local
102:    from all lookup main
103:    from all lookup vrf_wan
170:    from all fwmark 0x800000a9 lookup vrf_wan
254:    from all fwmark 0x800000fd lookup main
1000:   from all lookup [l3mdev-table]
2000:   from all lookup [l3mdev-table] unreachable
32765:  from all lookup local
32766:  from all lookup main
32767:  from all lookup default

The kernel will start from the lowest rule. If the rule matches source, destination or fwmark, it will look up in the table. If there is a matching entry, the process stops and the route is chosen. If the table does not contain a matching entry, the next rule is processed and so on (please correct me if I am wrong with this).

What now if there is a VRF that's assigned to a specific interface and/or process? Is this completely orthogonal to ip rule show? Do I need to choose between them? Or can both be used at the same time?

Remark: My system is VyOS. Rules 1000, 2000, 32765-32767 are generated by VyOS and can't be removed. Rules 170 & 254 are also generated by policy rules in VyOS but are overwritten by my other policies (rules 101-103).

4
  • Can you explain why the routing rules entries 101 and 102 were added, since they cancel the work of routing rule 2000 and the move of local to 32765? Is that the actual routing rules, or just a test?
    – A.B
    Commented Jun 6, 2023 at 7:55
  • @A.B Yes so this is actually because I am not generating the entries directly. I am using VyOS and in its default config there are just tables 1000, 2000, 32765-32767. These are at the very end. But I do want to have local and main (without default route!) searched FIRST and then check my own one (vrf_wan, containing the default route). So I added explicit rules that look them up earlier. Makes sense? Having said that, my question is general and not necessarily related to this exact instance.
    – divB
    Commented Jun 6, 2023 at 21:43
  • Thank you, your answer answers my question. But you emphasize my setup is "strange". Can you say why? As I mentioned, rules 1000, 2000, 32765-32767 are done by VyOS ... I can't remove them. So I have to "overwrite" them by creating rules with lower priority. I want local interfaces to be highest priority ("lookup local"), then consult all static+OSPF routes ("lookup main") and finally chose the default route. In this case, there is only one default route ("lookup vrf_wan") but more will be added depending on source address.
    – divB
    Commented Jun 7, 2023 at 17:03
  • I just missed the fact that the main routing table has no default route. And as you stated rules pref 170 and 254 (which appear redundant since they just have an additional selector compared to 102 and 103) are also created by VyOS, it makes sense now. But yet, I'm not sure ip exec vrf's behavior (which you appear to have a problem with) is easy to predict with this setup since it could match routes in table main (depending on its content) rather than its dedicated table. With VyOS' default setup, ip vrf exec ... can only match routes in the vrf's table.
    – A.B
    Commented Jun 7, 2023 at 18:16

1 Answer 1

2

The VRF has by default an associated routing rule with priority 1000 as described in VRF documentation below and as seen in OP's case:

  1. An l3mdev FIB rule directs lookups to the table associated with the device. A single l3mdev rule is sufficient for all VRFs. The VRF device adds the l3mdev rule for IPv4 and IPv6 when the first device is created with a default preference of 1000. Users may delete the rule if desired and add with a different priority or install per-VRF rules.

So you can delete the priority 1000 rules and create manual rules or just adjust your own rules: pref < 1000 or pref > 1000 depending on the need, as you did.

Additional information on VRF can be found in this blog: Stefan's Blog Using VRF (Virtual Routing and Forwarding) on Linux, including why VyOS creates rule pref 2000 and why the local table lookup is moved from pref 0 to pref 32765.


UPDATE: addressing OP's comment.

What does ip vrf exec blue ...? It set ups a BPF filter that forces a socket option whenever the affected process(es) creates a socket:

[...]

BPF filter attached to vrf/NAME cgroup2 to set sk_bound_dev_if to the VRF device index. [...] In doing so all AF_INET/AF_INET6 (ipv4/ipv6) sockets are automatically bound to the VRF domain.

I understand it's equivalent to a VRF-aware process using setsockopt(fd, SOL_SOCKET, SO_BINDTODEVICE, "vrf-blue", ...).

That meansoif rules or from what I understand l3mdev rules, will specifically be selected by this (if they get to be used before a route would be resolved before), else only remaining routes matching the device will match.

(Also from experience (outside of VRF) I know if there's no route for a destination when bound to an interface it behave as if a route was added to be able to emit a packet when binding, but this route won't have any gateway so would rarely work correctly).

Note that this was easier to understand before kernel 4.8 where one had to systematically do:

Prior to the v4.8 kernel iif and oif rules are needed for each VRF device:

ip ru add oif vrf-blue table 10
ip ru add iif vrf-blue table 10

The oif rule has a role if one tries to short-curcuit the rule in pref 1000 with standard rules (without l3mdev).

1
  • One followup question (maybe you can comment in your answer): What happens if a process is bound to an VRF (e.g. via ip vrf exec) and what happens if not? The link suggests a VRF really just calls a rule (so VRF is based on ip rule) but I am seeing an odd issue on my system that suggests differently.
    – divB
    Commented Jun 7, 2023 at 7:22

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .