Unfortunately even different portions of the same build may be optimal with conflicting j factor values, depending on what's being built, how, which of the system resources are the bottleneck at that time, what else is happening on the build machine, what's going on in the network (if using distributed build techniques), status/location/performance of the many caching systems involved in a build, etc.
Compiling 100 tiny C files may be faster than compiling a single huge one, or viceversa. Building small highly convoluted code can be slower than building huge amounts of straight-forward/linear code.
Even the context of the build matters - using a j factor optimized for builds on dedicated servers fine tuned for exclusive, non-overlapping builds may yield very dissapointing results when used by developers building in parallel on the same shared server (each such build may take more time than all of them combined if serialized) or on servers with different hardware configurations or virtualized.
There's also the aspect of correctness of the build specification. Very complex builds may have race conditions causing intermittent build failures with occurence rates that can vary wildly with the increase or decrease of the j factor.
I can go on and on. The point is that you have to actually evaluate your build in your very context for which you want the j factor optimized. @Jeff Schaller's comment applies: iterate until you find your best fit. Personally I'd start from the nproc value, try upwards first and downwards only if the upwards attempts show immediate degradation.
Might be a good idea to first measure several identical builds in supposedly identical contexts just to get an idea of the variability of your measurements - if too high it could jeopardise your entire optimisation effort (a 20% variability would completely eclipse a 10% improvement/degradation reading in the j factor search).
Lastly, IMHO it's better to use an (adaptive) jobserver if supported and available instead of a fixed j factor - it consistently provides a better build performance across wider ranges of contexts.