3

We need to configure NIC for better performance of our application on FreeBSD 12. System is FreeBSD 12.0-RELEASE-p3 GENERIC amd64

To start, I would like to show our solution for Linux system. For setup 16 RX and 16 TX queues on our multi-queue NIC we apply command ethtool -L ens786f1 combined 16. We are looking for alternative call for FreeBSD system.

On FreeBSD, our NIC is driven by ixl driver. I have found some kernel variables that should do the same thing like override_qs_enable, override_nrxqs and override_ntxqs (here is the manpage). Command to set them would look like sysctl dev.ixl.1.iflib.override_qs_enable="1" and so on (ens786f1 and ixl1 is the same device with different interface on different system).

I would like to set these variables every time on boot. Manpage says that these variables have to be set on boot before the ixl driver is loaded (using loader), so I added setup lines into /boot/loader.conf. The file was empty before, so now it looks like:

#Setup NIC#
dev.ixl.1.iflib.override_qs_enable="1"
dev.ixl.1.iflib.override_nrxqs="16"
dev.ixl.1.iflib.override_ntxqs="16"

/boot/defaults/loader.conf included /boot/loader.conf using loader_conf_files, so my boot file should be running. But when I reboot and look at the variables using sysctl, these variables are default (0) and when I look at dmesg, device is not setup the way I wanted (still shows 8 RX and 8 TX queues).

It's possible that /boot/loader.conf is not called; maybe it's just bad approach to replace ethtool command on FreeBSD.

Some logs: pastebin.com/YGTxqxdx

I have also tried to set up variables using kenv (save in file also) but it seems that options aren't saved after a reboot.

After few days of research, I have found out that I probably have to set up new / edit old VF of my NIC using iovctl. When I call iovctl -S -d ixl1, there is a parameter num-queues. Does anyone have experience with this configuration?

Problem was with Loader - wrong loader setup in GRUB. Still, if you are looking for help regarding setting up a number of RX/TX queues with same driver, the answer below is a really good set of information.

Improve this question
5
  • 1
    Your first approach seems correct for me. iovctl is however for setting SR-IOV. The ixl driver supports this but is for I/O virtualization and allows the card to appear as multiple device. See What is SR-IOV?.
    – Claus Andersen
    Commented Apr 10, 2019 at 14:00
  • The card does appear as multiple device (ixl0 and ixl1) becouse it has more ports. But right now I'm totally desperate, so I'm trying any solution..
    – Payne
    Commented Apr 10, 2019 at 14:21
  • 1
    You should add the output of sysctl dev.ixl hw.ixl hw.intr_storm_threshold, sysctl -d dev.ixl hw.ixl and dmesg | grep ^ixl to your question. Reading the ixl man page I am not sure that this driver uses iflib. Maybe it is hw.ixl.max_queues you are looking for? Are you even sure you need to set the queues? The defaults might be OK on FreeBSD as the driver is not the same as on Linux. You conflate 3 problems into 1: a) Fix a (unknown) performance issue b) Finding the correct tunable setting and c) making it persist.
    – Claus Andersen
    Commented Apr 10, 2019 at 14:42
  • I post logs on pasterbin, becouse they were realy long.. Im not sure where I obtain information that iflib is ixls driver interface.. There is no hw.ixl.max_queues variable to setup. In short, I'm looking for solution for adding more rxq/txq descriptors. And yes, I need to find correct tunables and making them persist. But find the currect tunables is the main problem, making them persist should be secondary problem. None fixing of performance issue, just tunning..
    – Payne
    Commented Apr 11, 2019 at 7:04
  • it would be really helpful if you added another answer explaining what setting you had in Grub - and whatever you did to fix it. It is nice with your short update in your question. But a full answer would be excellent. A question can have multiple fulfilling answers. Cheers.
    – Claus Andersen
    Commented Apr 15, 2019 at 9:30

1 Answer 1

Reset to default
3

See How to make setting of kern.ipc.somaxconn persistent?

Remove quotes "

dev.ixl.1.iflib.override_qs_enable=1
dev.ixl.1.iflib.override_nrxqs=16
dev.ixl.1.iflib.override_ntxqs=16

Test with /etc/rc.d/sysctl start or service sysctl restart

Update:

In earlier versions of FreeBSD you would normally set hw.ixl.max_queues. But it seems that the ixl driver changed to using iflib with FreeBSD 12.

After sysctl -d dev.ixl we can indeed confirm that the driver is using iflib

dev.ixl.1.iflib.override_qs_enable: permit #txq != #rxq
dev.ixl.1.iflib.override_nrxqs: # of rxqs to use, 0 => use default #
dev.ixl.1.iflib.override_ntxqs: # of txqs to use, 0 => use default #

A web search confirms this as we can find the diff which adds it to the driver (rS335338: ixl(4): Update to use iflib).

The settings are currently as default:

dev.ixl.1.iflib.override_qs_enable: 0
dev.ixl.1.iflib.override_nrxqs: 0
dev.ixl.1.iflib.override_ntxqs: 0

This should the set the number of queues to the number of cores (In this case 8). The iflib man page states that these must be set before loading the driver. Sysctl restart will then not help us - only reboots.

We must then set /boot/loader.conf:

dev.ixl.1.iflib.override_qs_enable=1
dev.ixl.1.iflib.override_nrxqs=16
dev.ixl.1.iflib.override_ntxqs=16

After setting this you should reboot and verify with dmesg | grep ^ixl and sysctl dev.ixl.1.iflib.

Your current output from dmesg was:

ixl0: pxm cpus: 8 queue msgs: 128 admincnt: 1
ixl0: using 8 rx queues 8 tx queues

ixl1: pxm cpus: 8 queue msgs: 128 admincnt: 1
ixl1: using 8 rx queues 8 tx queues

You claim you have done all this before to no avail.

If we look in the bug report for Bug 230465 - ixl: not working in netmap mode we do however get confirmation from Jeff Pieper. Charles Goncalves further confirms that it works for him.

This is a rather typical way of setting things in FreeBSD and usually works. As always it could be a buggy driver but with the above confirmation I think not. I do not have an ixl card so unfortunately I cannot confirm anything.

You mention kenv as well. And Jeff Pieper does confirm this can be set. But note you can only use kenv when working with a driver module and not the static driver in the kernel!

So with the above I would claim that it "ShouldWork(tm)" with settings in loader.conf with the kernel driver in FreeBSD 12 (or later). And it is a reminder to almost always add uname -a to any FreeBSD related questions 😉.

Desperate times - desperate measures

You also reported dev.ixl.1.iflib.driver_version: 2.1.0-k. I assume this is the static kernel version of the driver. According to the ixl man page the upstream driver from Intel is updated more frequently and is supplied in net/intel-ixl-kmod.

If you want to go this route - then make sure that you have populated /usr/src and have an up-to-date ports tree. There is not a pre-built package available.

To install the kmod driver:

cd /usr/ports/net/intel-ixl-kmod/ && make install clean

Make sure /boot/loader.conf looks like this:

dev.ixl.1.iflib.override_qs_enable=1
dev.ixl.1.iflib.override_nrxqs=16
dev.ixl.1.iflib.override_ntxqs=16
if_ixl_updated_load="YES"

Reboot and see effects in dmesg and sysctl.

When using this modular driver you can also set the environment dynamicly using kenv as Jeff Pieper described. Unload and load the driver using kldunload and kldload

Interrupt storms

I noticed that you had:

hw.intr_storm_threshold: 1000

But as it is only a X710 for 10GbE it is probably not a problem. According to the ixl man page you might see interrupt storms at 40GbE. Then you would set:

/etc/sysctl.conf:

 hw.intr_storm_threshold=0

(hence the interest in this parameter)

Firmware

And lastly I noticed:

dev.ixl.1.fw_version: fw 5.0.40043 api 1.5 nvm 5.05 etid 80002927 oem 1.261.0

That firmware is more than two years old. There is a version 6.80 available for the Intel(R) Ethernet Controller X710

The easy route is to go with sysutils/intel-nvmupdate. In can be installed quickly as a binary package (no need to build).

Improve this answer
4
  • I did remove quotes but it seems that changes are still not persistent.. On boot nor in /var/log/console.log I have no warning about wrong setup of there variables. But I think I'm on the right path now, watch EDIT #2 in my original question..
    – Payne
    Commented Apr 10, 2019 at 12:50
  • Did you see the change when restarting sysctl? And do you have multiple cards? The first card is 0 (ixl0)
    – Claus Andersen
    Commented Apr 10, 2019 at 13:42
  • When I do sysctl restart, no change happens.. I do have multiple cards (exactly 2), both with 2 ports.. ifconfig shows ix0, ix1, ixl0, ixl1 - ixl1 is the one I need to setup for sure
    – Payne
    Commented Apr 10, 2019 at 14:16
  • 1
    Thank you very much for your help.. As you mentioned, this setup should help. So I focused on the loading of variables, and find out, that in GRUB I had wrong setup of loader, so loader.conf file had no efect..
    – Payne
    Commented Apr 12, 2019 at 8:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .