1

Moved from here since this is the correct StackExchange.

Goal

  • Recompile the NVMe driver on Rocky Linux 9 without any changes (I want to make some but right now I'm just trying to get a working module)

UPDATE TO THE BELOW

  1. I figured out what was causing the symbol error. That was coming from not loading nvme-common first. nvme-core depends on nvme-common so you have to load it first and doing so gets me past that error.
  2. Now I have the same problem as this question. Everything points to some discrepency between the kernel versions but just as that person indicated, everything I can find matches.

My new setup is:

dnf install -y rpm-build rpmdevtools git python3-devel make gcc flex bison kernel-headers ncurses-devel tmux elfutils-libelf-devel openssl-devel bc kernel-devel-$(uname -r) dwarves
rpmdev-setuptree
dnf download --source kernel
rpm -ivh kernel-5.14.0-362.18.1.el9_3.src.rpm
rpmbuild -bp kernel.spec
cd /root/rpmbuild/BUILD/kernel-5.14.0-362.18.1.el9_3/linux-5.14.0-362.18.1.el9.x86_64/
cp -f /boot/config-$(uname -r) .config
cp -f /usr/src/kernels/$(uname -r)/Module.symvers .
make -j$(nproc --all) scripts prepare modules_prepare
make ARCH=x86_64 -j$(nproc --all) M=drivers/nvme
insmod drivers/nvme/common/nvme-common.ko
insmod drivers/nvme/host/nvme-core.ko

Doing it that way also makes the dependencies match perfectly.

What I'm doing

  1. Update everything (dnf update -y && reboot) and reboot to make sure it's all where it needs to be
  2. Install headers dnf install -y kernel-headers ncurses-devel
  3. Get kernel source dnf download --source kernel && rpm2cpio kernel-5.14.0-362.18.1.el9_3.src.rpm | cpio -idmv && tar -xf linux-5.14.0-362.18.1.el9_3.tar.xz (this is where I'm getting the NVMe source code)
  4. Pull in Module.symvers from the headers cp /usr/src/kernels/5.14.0-362.18.1.el9_3.x86_64/Module.symvers .
  5. Pull in the config from my specific kernel cp /boot/config-$(uname -r) .config
  6. Run the following to build:
make clean
cp -f /boot/config-$(uname -r) .config
cp -f /usr/src/kernels/5.14.0-362.18.1.el9_3.x86_64/Module.symvers .
make -j$(nproc --all) scripts prepare modules_prepare
make ARCH=x86_64 -j$(nproc --all) M=drivers/nvme

The Problem

If I leave the NVMe options to default:

enter image description here

and then build it produces this error:

[23890.056877] nvme_core: disagrees about version of symbol nvme_auth_gen_shared_secret
[23890.056881] nvme_core: Unknown symbol nvme_auth_gen_shared_secret (err -22)
[23890.057140] nvme_core: disagrees about version of symbol nvme_auth_gen_pubkey
[23890.057142] nvme_core: Unknown symbol nvme_auth_gen_pubkey (err -22)
[23890.057471] nvme_core: disagrees about version of symbol nvme_auth_gen_privkey
[23890.057472] nvme_core: Unknown symbol nvme_auth_gen_privkey (err -22)

If I update the options to remove all the stuff I don't care about I receive a different error:

enter image description here

module: x86/modules: Skipping invalid relocation target, existing value is nonzero for type 1, loc 0000000087bc08cc, val ffffffffc071647a

I'm at a total loss though. Everything online says the cause of this is different kernel versions, but I cannot figure out where. Every resource seems to say something about making sure .config and Module.symvers match which I have done.

Here is what the two modules modified/original look like compared:

[root@nvmetest linux-5.14.0-362.18.1.el9_3]# !mod
modinfo ./drivers/nvme/host/nvme-core.ko
filename:       /root/new_driver/linux-5.14.0-362.18.1.el9_3/./drivers/nvme/host/nvme-core.ko
version:        1.0
license:        GPL
rhelversion:    9.3
srcversion:     A869617A5E58420845515F4
depends:        t10-pi
retpoline:      Y
name:           nvme_core
vermagic:       5.14.0 SMP preempt mod_unload modversions
parm:           multipath:turn on native support for multiple controllers per subsystem (bool)
parm:           iopolicy:Default multipath I/O policy; 'numa' (default) or 'round-robin'
parm:           admin_timeout:timeout in seconds for admin commands (uint)
parm:           io_timeout:timeout in seconds for I/O (uint)
parm:           shutdown_timeout:timeout in seconds for controller shutdown (byte)
parm:           max_retries:max number of retries a command may have (byte)
parm:           default_ps_max_latency_us:max power saving latency for new devices; use PM QOS to change per device (ulong)
parm:           force_apst:allow APST for newly enumerated devices even if quirked off (bool)
parm:           apst_primary_timeout_ms:primary APST timeout in ms (ulong)
parm:           apst_secondary_timeout_ms:secondary APST timeout in ms (ulong)
parm:           apst_primary_latency_tol_us:primary APST latency tolerance in us (ulong)
parm:           apst_secondary_latency_tol_us:secondary APST latency tolerance in us (ulong)
[root@nvmetest linux-5.14.0-362.18.1.el9_3]# modinfo /lib/modules/5.14.0-362.18.1.el9_3.x86_64/kernel/drivers/nvme/host/nvme-core.ko.xz
filename:       /lib/modules/5.14.0-362.18.1.el9_3.x86_64/kernel/drivers/nvme/host/nvme-core.ko.xz
version:        1.0
license:        GPL
rhelversion:    9.3
srcversion:     ADFE53FFFB5D30ECFF130B0
depends:        nvme-common,t10-pi
retpoline:      Y
intree:         Y
name:           nvme_core
vermagic:       5.14.0-362.18.1.el9_3.x86_64 SMP preempt mod_unload modversions
sig_id:         PKCS#7
signer:         Rocky kernel signing key
sig_key:        37:B0:46:2C:D4:62:CB:E7:6C:CA:AE:9F:2A:A2:BE:E1:36:3A:8A:AF
sig_hashalgo:   sha256
signature:      60:89:BA:1D:1C:71:38:82:DF:09:73:B4:23:3E:C8:FE:7B:E4:9F:0D:
                62:6E:28:D7:3E:5A:5E:11:CD:7B:D2:52:E2:C6:ED:5E:B6:A7:19:54:
                8A:FB:BB:E8:2D:A5:77:3F:A1:C1:7E:EB:45:74:30:E9:18:1C:3D:9A:
                53:4A:2B:B0:1E:F0:35:D3:D1:E5:B6:A5:D0:47:6C:2F:B7:C6:6F:00:
                30:0E:82:BA:FD:4F:9D:0E:3B:4A:17:A4:1B:E8:31:FC:FB:BC:C2:93:
                1C:6D:5E:94:FD:DE:65:3B:3E:0B:F4:B4:B0:82:67:87:8C:90:C9:74:
                44:BB:14:D9:F9:43:33:CC:CC:77:29:11:2C:3D:79:30:EA:B3:63:74:
                F3:02:F0:DA:68:40:BA:65:B0:E5:D8:90:FF:B0:CA:8D:D7:31:00:47:
                FE:9C:B9:17:8F:81:1D:7F:45:F6:98:E8:14:1F:73:99:00:51:18:48:
                1F:29:98:F4:37:FA:62:46:FF:1B:64:B5:1F:03:C3:5C:87:2E:13:9E:
                EE:8C:32:DE:D8:B6:3F:1D:C2:69:45:46:E2:8B:E4:BD:C2:7C:00:14:
                3F:7B:76:C8:43:4E:ED:24:BE:C8:9D:85:16:C6:9B:55:1F:BA:7B:39:
                07:57:A7:46:1A:E4:98:D5:29:C9:27:07:0B:3A:FE:6D:49:4B:DD:24:
                E0:4C:99:C1:C4:88:4D:E1:D9:78:EC:46:4F:D6:94:D6:93:B0:D4:24:
                23:08:40:35:F9:41:0D:1E:4A:78:3C:B2:A9:DB:51:C9:D0:96:F5:64:
                43:7E:FF:69:71:09:06:9D:79:B0:56:A0:49:71:69:64:3D:50:B6:0B:
                DE:A0:FA:36:D7:86:AD:B9:2A:8C:11:B5:73:F3:C5:2B:C5:2F:C2:A6:
                AB:7F:01:B1:E6:60:8F:6A:F0:A1:AE:A9:32:E1:30:DA:4D:7A:98:F5:
                89:F7:7B:4D:FF:23:4B:77:29:BA:62:1B:54:09:1F:33:57:8F:44:A3:
                FE:19:D6:43
parm:           multipath:turn on native support for multiple controllers per subsystem (bool)
parm:           iopolicy:Default multipath I/O policy; 'numa' (default) or 'round-robin'
parm:           admin_timeout:timeout in seconds for admin commands (uint)
parm:           io_timeout:timeout in seconds for I/O (uint)
parm:           shutdown_timeout:timeout in seconds for controller shutdown (byte)
parm:           max_retries:max number of retries a command may have (byte)
parm:           default_ps_max_latency_us:max power saving latency for new devices; use PM QOS to change per device (ulong)
parm:           force_apst:allow APST for newly enumerated devices even if quirked off (bool)
parm:           apst_primary_timeout_ms:primary APST timeout in ms (ulong)
parm:           apst_secondary_timeout_ms:secondary APST timeout in ms (ulong)
parm:           apst_primary_latency_tol_us:primary APST latency tolerance in us (ulong)
parm:           apst_secondary_latency_tol_us:secondary APST latency tolerance in us (ulong)
[root@nvmetest linux-5.14.0-362.18.1.el9_3]#

I can force the version magic to be completely the same using CONFIG_LOCALVERSION, but the resulting errors are identical. This post also indicates that the base kernel version is sufficient.

4
  • Look at the depends.... What's the difference between DEPENDS: 10t-pi and DEPENDS: nvme-common, t10-pi other than the fact that it's there when both modules are named nvme_core? Shouldn't both configs be identical before you start? Acording to CONFIG_NVME_COMMON doesn't exist in the 5.x branch You need a 6.x source, which means you need a 6.x kernel, and I haven't used ROCK to know if they have one
    – eyoung100
    Commented Feb 5 at 15:08
  • Interesting... so I haven't pulled my NVMe source externally. I'm using the source that's in the folder that came with dnf download --source kernel && rpm2cpio kernel-5.14.0-362.18.1.el9_3.src.rpm | cpio -idmv && tar -xf linux-5.14.0-362.18.1.el9_3.tar.xz. Will investigate. Commented Feb 5 at 15:13
  • Let me know. Compiling kernel sources on Gentoo, my main distribution, is commonplace for me but we will get that NVMe device up and running
    – eyoung100
    Commented Feb 5 at 15:17
  • See my answer below. Adding an extra repository might be faster and safer than compiling the entire kernel from scratch.
    – eyoung100
    Commented Feb 5 at 18:07

2 Answers 2

1

After nuking the kernel when I tried to depmod things despite the warnings, I rebuilt. After a rebuild I did the following:

Confirm there are no updates available. If there are, reboot the box after updating. Then:

dnf install -y rpm-build rpmdevtools git python3-devel make gcc flex bison kernel-headers ncurses-devel tmux elfutils-libelf-devel openssl-devel bc kernel-devel-$(uname -r) dwarves
rpmdev-setuptree
dnf download --source kernel
rpm -ivh kernel-5.14.0-362.18.1.el9_3.src.rpm
cd /root/rpmbuild/SPECS
rpmbuild -bp kernel.spec
cd /root/rpmbuild/BUILD/kernel-5.14.0-362.18.1.el9_3/linux-5.14.0-362.18.1.el9.x86_64/
cp -f /boot/config-$(uname -r) .config
cp -f /usr/src/kernels/$(uname -r)/Module.symvers .
make -j$(nproc --all) scripts prepare modules_prepare
make ARCH=x86_64 -j$(nproc --all) M=drivers/nvme
insmod drivers/nvme/common/nvme-common.ko
insmod drivers/nvme/host/nvme-core.ko

I have no idea what the delta is. I checked at least a dozen times a dozen different ways my kernel source matched the running kernel previously, but obviously I missed something. On rebuild the above worked perfectly the first try.

1
  • When in doubt, reboot, also known as start over. I'm glad your original method of attack/approach worked out. I have to ask though: What do the DEPENDS look like now?
    – eyoung100
    Commented Feb 6 at 16:05
0

Solution

After seeing the extra DEPENDS above per my comment, I looked it up at The Linux Kernel Driver Database: CONFIG_NVME_COMMON. According to that link, that option is only available starting in Kernel version 6. That led me to write this answer, which starts with this question: How can we get a 6.x branch Kernel on a RedHat/Fedora based distribution when it isn't in the repository?

We have 2 options:

  1. We can build our own.
  2. We can add the Enterprise Linux repository, where a version of the 6.x kernel is available, and possibly the OP's needed NVMe driver.

Since Rocky is a Fork of CentOS, and CentOS is a fork of RedHat's Enterprise Linux, #2 seems the ideal choice. Quoting their homepage:

The ELRepo Project focuses on hardware related packages to enhance your experience with Enterprise Linux. This includes SCSI/SATA/PATA drivers, filesystem drivers, graphics drivers, network drivers, sound drivers and video drivers.

What that quote fails to mention is that the repository also contains newer kernels. I was going to manually write the steps needed to obtain that kernel, but in the interest of brevity, and in hopes the driver the OP needs is also in that repository, I found multiple links with those directions:

As you can see, the steps are identical in both links. I'll add them at some point if we start having a case of Link Rot. After upgrading the kernel from the elrepo-kernel repository, per either link, I would then search the EL repo to see if the OP's driver is also available. If it isn't the OP is free to use option 1 offered above, which is I believe where he started to begin with.

3
  • I'm still working on it. It's not a kernel version problem. What happens is that with Rocky Linux you have to apply patches. Instead of using cpio to extract everything you have to run rpm -ivh kernel-5.14.0-362.18.1.el9_3.src.rpm to install the patches which then makes the dependencies match. However, that doesn't actually matter. The symbol problem came from the fact that you have to load nvme-common FIRST and then nvme-core. nvme-common has the symbols nvme-core was looking for. Commented Feb 5 at 19:36
  • Ahh, then I suppose that Rocky Backported that from the Vanilla 6.x kernel. I'd be curious to find out. Let me know here either way.
    – eyoung100
    Commented Feb 5 at 19:40
  • See my answer above. I don't like it, but basically what I was doing originally should have worked and I haven't the vaguest idea why it didn't. I've gone back through and checked the outputs from modinfo I have from today against what I have from yesterday and they are identical aside from source version. Something changed in the source code from yesterday to today, but I haven't the vaguest idea what it was. Commented Feb 6 at 15:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .