2

I have amulti-threaded server application running in Linux 4.4.0 with a X540-AT2 NIC (the server has one thread per core). Since Linux has RSS enabled, it uses one NIC receive queue for each core in the system (16 cores and therefore 16 RX queues).

My objective is to let a client application running on a separate host to "hint" to what queue should packets be directed (i.e., some kind of client-directed receive queue load balancing).

To achieve this, I've been playing with the NIC's flow director table with no luck (any ideas?):

  • VLAN tag: The server host assigns each VLAN identifier to a separate RX queue (using ethtool --config-ntuple), and the client app assigns a VLAN tag to each packet in order to identify the target receive queue (therefore achieving the client-directed balancing I want).

    Unfortunately, the server never receives any of the client's packets, since it is listening on the main NIC interface, which has no VLANs assigned to it in the system. Is there a way to drop the VLAN tags once packets are received, so that they'll only be used for my balancing needs?

  • TOS field (bits 8-15 of IPv4 header): I've also tried using IPv4's TOS field to do the same. Te server host uses ethtool to direct each TOS value to a separate queue, and the client crafts sent packets to have a TOS value according to the desired target receive queue on the server.

    Unfortunately, it seems that ethtool is ignoring my TOS values on the filters (TOS is always 0 on the rules):

    $ sudo ethtool -U em2 flow-type tcp4 tos 1 action 10
    Added rule with ID 2045
    $ ethtool --show-ntuple em2
    16 RX rings available
    Total 1 rules
    Filter: 2045
            Rule Type: TCP over IPv4
            Src IP addr: 0.0.0.0 mask: 255.255.255.255
            Dest IP addr: 0.0.0.0 mask: 255.255.255.255
            TOS: 0x0 mask: 0xff
            Src port: 0 mask: 0xffff
            Dest port: 0 mask: 0xffff
            VLAN EtherType: 0x0 mask: 0xffff
            VLAN: 0x0 mask: 0xffff
            User-defined: 0x0 mask: 0xffffffffffffffff
            Action: Direct to queue 10
    
  • user-def: I've also tried with user-def to overcome the TOS field "problem", but it seems I can only use that on the last two bytes:

    $ sudo ethtool -U em2 flow-type tcp4 user-def 2  action 10
    Added rule with ID 2045
    $ ethtool --show-ntuple em2
    16 RX rings available
    Total 1 rules
    Filter: 2045
            Rule Type: TCP over IPv4
            Src IP addr: 0.0.0.0 mask: 255.255.255.255
            Dest IP addr: 0.0.0.0 mask: 255.255.255.255
            TOS: 0x0 mask: 0xff
            Src port: 0 mask: 0xffff
            Dest port: 0 mask: 0xffff
            VLAN EtherType: 0x0 mask: 0xffff
            VLAN: 0x0 mask: 0xffff
            User-defined: 0x2 mask: 0xffffffffffffff00
            Action: Direct to queue 10
    

    And when I try to match some other bytes, it's simply ignored (user-defined is always zero and the mask is full):

    $ sudo ethtool -U em2 flow-type tcp4 user-def 2 m 0xf0ffffffffffffff  action 10
    Added rule with ID 2045
    $ ethtool --show-ntuple em2
    16 RX rings available
    Total 1 rules
    Filter: 2045
            Rule Type: TCP over IPv4
            Src IP addr: 0.0.0.0 mask: 255.255.255.255
            Dest IP addr: 0.0.0.0 mask: 255.255.255.255
            TOS: 0x0 mask: 0xff
            Src port: 0 mask: 0xffff
            Dest port: 0 mask: 0xffff
            VLAN EtherType: 0x0 mask: 0xffff
            VLAN: 0x0 mask: 0xffff
            User-defined: 0x0 mask: 0xffffffffffffffff
            Action: Direct to queue 10
    

Any idea how I can solve the problems above? (any of the VLAN or TOS approaches would work for me).

Edit: Clarified question as requested by @Hauke Laging.

0

You should make clear what kind of queues you want to balance and with which effect.

In addition you should mention what you can configure on the server. You mentioned VLAN. It seems you cannot configure VLAN on the server.

There are several possibilities to send packets to the same system in different ways:

  1. Using different TCP/UDP ports (if TCP/UDP is used). This can be done transparently to the application by using DNAT on both the client and the server.
  2. Using different IP addresses on the server. This can be done transparently to the application by using DNAT on both the client and the server.
  3. Creating several tunnel interfaces on client and server (ip link add type ipip) and sending the packets over several interfaces.
  4. Ingress queueing can be done with the ifb device and traffic shaping (based on the above "information").
  • I've just rewritten the question in a way that I hope is much clearer. Thanks! – Lluís Vilanova Oct 15 '17 at 16:02
  • Oh, and for the record, I can configure whatever necessary on both the client and server hosts. – Lluís Vilanova Oct 15 '17 at 17:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.