insmod fails - module does not match running kernel but module has been compiled against correct kernel

Copied from here since this is the correct StackExchange.

Goal

• Recompile the NVMe driver on Rocky Linux 9 without any changes (I want to make some but right now I'm just trying to get a working module)

What I'm doing

1. Update everything (dnf update -y && reboot) and reboot to make sure it's all where it needs to be
2. Install headers dnf install -y kernel-headers ncurses-devel
3. Get kernel source dnf download --source kernel && rpm2cpio kernel-5.14.0-362.18.1.el9_3.src.rpm | cpio -idmv && tar -xf linux-5.14.0-362.18.1.el9_3.tar.xz
4. Pull in Module.symvers from the headers cp /usr/src/kernels/5.14.0-362.18.1.el9_3.x86_64/Module.symvers .
5. Pull in the config from my specific kernel cp /boot/config-$(uname -r) .config
6. Run the following to build:

make cleanmake menuconfig # Skip this if you pulled the config from the systemmake -j$(nproc --all) modules_preparemake -j$(nproc --all) oldconfig && make -j$(nproc --all) preparemake ARCH=x86_64 -j$(nproc --all) M=drivers/nvme

The Problem

If I leave the NVMe options to default:

and then build it produces this error:

[23890.056877] nvme_core: disagrees about version of symbol nvme_auth_gen_shared_secret[23890.056881] nvme_core: Unknown symbol nvme_auth_gen_shared_secret (err -22)[23890.057140] nvme_core: disagrees about version of symbol nvme_auth_gen_pubkey[23890.057142] nvme_core: Unknown symbol nvme_auth_gen_pubkey (err -22)[23890.057471] nvme_core: disagrees about version of symbol nvme_auth_gen_privkey[23890.057472] nvme_core: Unknown symbol nvme_auth_gen_privkey (err -22)

If I update the options to remove all the stuff I don't care about I receive a different error:

module: x86/modules: Skipping invalid relocation target, existing value is nonzero for type 1, loc 0000000087bc08cc, val ffffffffc071647a

I'm at a total loss though. Everything online says the cause of this is different kernel versions, but I cannot figure out where. Every resource seems to say something about making sure .config and Module.symvers match which I have done.

Here is what the two modules modified/original look like compared:

[root@nvmetest linux-5.14.0-362.18.1.el9_3]# !modmodinfo ./drivers/nvme/host/nvme-core.kofilename:       /root/new_driver/linux-5.14.0-362.18.1.el9_3/./drivers/nvme/host/nvme-core.koversion:        1.0license:        GPLrhelversion:    9.3srcversion:     A869617A5E58420845515F4depends:        t10-piretpoline:      Yname:           nvme_corevermagic:       5.14.0 SMP preempt mod_unload modversionsparm:           multipath:turn on native support for multiple controllers per subsystem (bool)parm:           iopolicy:Default multipath I/O policy; 'numa' (default) or 'round-robin'parm:           admin_timeout:timeout in seconds for admin commands (uint)parm:           io_timeout:timeout in seconds for I/O (uint)parm:           shutdown_timeout:timeout in seconds for controller shutdown (byte)parm:           max_retries:max number of retries a command may have (byte)parm:           default_ps_max_latency_us:max power saving latency for new devices; use PM QOS to change per device (ulong)parm:           force_apst:allow APST for newly enumerated devices even if quirked off (bool)parm:           apst_primary_timeout_ms:primary APST timeout in ms (ulong)parm:           apst_secondary_timeout_ms:secondary APST timeout in ms (ulong)parm:           apst_primary_latency_tol_us:primary APST latency tolerance in us (ulong)parm:           apst_secondary_latency_tol_us:secondary APST latency tolerance in us (ulong)[root@nvmetest linux-5.14.0-362.18.1.el9_3]# modinfo /lib/modules/5.14.0-362.18.1.el9_3.x86_64/kernel/drivers/nvme/host/nvme-core.ko.xzfilename:       /lib/modules/5.14.0-362.18.1.el9_3.x86_64/kernel/drivers/nvme/host/nvme-core.ko.xzversion:        1.0license:        GPLrhelversion:    9.3srcversion:     ADFE53FFFB5D30ECFF130B0depends:        nvme-common,t10-piretpoline:      Yintree:         Yname:           nvme_corevermagic:       5.14.0-362.18.1.el9_3.x86_64 SMP preempt mod_unload modversionssig_id:         PKCS#7signer:         Rocky kernel signing keysig_key:        37:B0:46:2C:D4:62:CB:E7:6C:CA:AE:9F:2A:A2:BE:E1:36:3A:8A:AFsig_hashalgo:   sha256signature:      60:89:BA:1D:1C:71:38:82:DF:09:73:B4:23:3E:C8:FE:7B:E4:9F:0D:                62:6E:28:D7:3E:5A:5E:11:CD:7B:D2:52:E2:C6:ED:5E:B6:A7:19:54:                8A:FB:BB:E8:2D:A5:77:3F:A1:C1:7E:EB:45:74:30:E9:18:1C:3D:9A:                53:4A:2B:B0:1E:F0:35:D3:D1:E5:B6:A5:D0:47:6C:2F:B7:C6:6F:00:                30:0E:82:BA:FD:4F:9D:0E:3B:4A:17:A4:1B:E8:31:FC:FB:BC:C2:93:                1C:6D:5E:94:FD:DE:65:3B:3E:0B:F4:B4:B0:82:67:87:8C:90:C9:74:                44:BB:14:D9:F9:43:33:CC:CC:77:29:11:2C:3D:79:30:EA:B3:63:74:                F3:02:F0:DA:68:40:BA:65:B0:E5:D8:90:FF:B0:CA:8D:D7:31:00:47:                FE:9C:B9:17:8F:81:1D:7F:45:F6:98:E8:14:1F:73:99:00:51:18:48:                1F:29:98:F4:37:FA:62:46:FF:1B:64:B5:1F:03:C3:5C:87:2E:13:9E:                EE:8C:32:DE:D8:B6:3F:1D:C2:69:45:46:E2:8B:E4:BD:C2:7C:00:14:                3F:7B:76:C8:43:4E:ED:24:BE:C8:9D:85:16:C6:9B:55:1F:BA:7B:39:                07:57:A7:46:1A:E4:98:D5:29:C9:27:07:0B:3A:FE:6D:49:4B:DD:24:                E0:4C:99:C1:C4:88:4D:E1:D9:78:EC:46:4F:D6:94:D6:93:B0:D4:24:                23:08:40:35:F9:41:0D:1E:4A:78:3C:B2:A9:DB:51:C9:D0:96:F5:64:                43:7E:FF:69:71:09:06:9D:79:B0:56:A0:49:71:69:64:3D:50:B6:0B:                DE:A0:FA:36:D7:86:AD:B9:2A:8C:11:B5:73:F3:C5:2B:C5:2F:C2:A6:                AB:7F:01:B1:E6:60:8F:6A:F0:A1:AE:A9:32:E1:30:DA:4D:7A:98:F5:                89:F7:7B:4D:FF:23:4B:77:29:BA:62:1B:54:09:1F:33:57:8F:44:A3:                FE:19:D6:43parm:           multipath:turn on native support for multiple controllers per subsystem (bool)parm:           iopolicy:Default multipath I/O policy; 'numa' (default) or 'round-robin'parm:           admin_timeout:timeout in seconds for admin commands (uint)parm:           io_timeout:timeout in seconds for I/O (uint)parm:           shutdown_timeout:timeout in seconds for controller shutdown (byte)parm:           max_retries:max number of retries a command may have (byte)parm:           default_ps_max_latency_us:max power saving latency for new devices; use PM QOS to change per device (ulong)parm:           force_apst:allow APST for newly enumerated devices even if quirked off (bool)parm:           apst_primary_timeout_ms:primary APST timeout in ms (ulong)parm:           apst_secondary_timeout_ms:secondary APST timeout in ms (ulong)parm:           apst_primary_latency_tol_us:primary APST latency tolerance in us (ulong)parm:           apst_secondary_latency_tol_us:secondary APST latency tolerance in us (ulong)[root@nvmetest linux-5.14.0-362.18.1.el9_3]#

I can force the version magic to be completely the same using CONFIG_LOCALVERSION, but the resulting errors are identical. This post also indicates that the base kernel version is sufficient.