There are several things you could do:
Block kernel module loading until the system is rebooted
echo 1 > /proc/sys/kernel/modules_disabled
After this, no new modules can be loaded for as long as the kernel is running. This setting cannot be reset back to 0 without rebooting.
This still allows loading modules at boot time, but allows you to lock it down once all the necessary modules have been loaded.
Ensure that only trustworthy modules can be loaded in the first place
These methods will allow you to have your cake and eat it too:
If your system has UEFI firmware and Secure Boot is enabled, it is actually a Secure Boot certification requirement that the bootloader must not allow unsigned kernel code to be executed. Most distributions that support Secure Boot will extend this to kernel modules too, using kernel module signing (kernel compile options
Or, if you compile your own kernel, you could enable the
CONFIG_SECURITY_LOADPIN kernel compilation option, which adds a requirement that all kernel modules must come from a single filesystem. This could be useful if you've segregated all user-writeable and temporary directories to separate filesystems, or even have your root filesystem read-only in locked-down use.
And of course, there is the hard-core option...
Build your own custom kernel with all the necessary drivers built-in and the module functionality entirely disabled
This is old-school, but can still be applicable if your hardware configuration is very stable.