Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It only takes a minute to sign up.
Sign up to join this community
I want to compile as fast as possible. Go figure. And would like to automate the choice of the number following the -j option. How can I programmatically choose that value, e.g. in a shell script?
Is the output of nproc equivalent to the number of threads I have available to compile with?
make -j1
make -j16
nproc gives the number of CPU cores/threads available, e.g. 8 on a quad-core CPU supporting two-way SMT.
The number of jobs you can run in parallel with make using the -j option depends on a number of factors:
make jobmake jobs are I/O- or CPU-boundmake -j$(nproc) is a decent place to start, but you can usually use higher values, as long as you don't exhaust your available memory and start thrashing.
For really fast builds, if you have enough memory, I recommend using a tmpfs, that way most jobs will be CPU-bound and make -j$(nproc) will work as fast as possible.
tmpfs, will I be limited to a directory size that always smaller than my physical RAM size?
time call. Clean up the results, lather rinse repeat-- and end up sorting the times/j values.
The most straight-foward way is to use nproc like so:
make -j`nproc`
The command nproc will return the number of cores on your machine. By wrapping it in the ticks, the nproc command will execute first, return a number and that number will be passed into make.
You may have some anecdotal experience where doing core-count + 1 results in faster compile times. This has more to do with factors like I/O delays, other resource delays and other availability of resource constraints.
To do this with nproc+1, try this:
make -j$((`nproc`+1))
Unfortunately even different portions of the same build may be optimal with conflicting j factor values, depending on what's being built, how, which of the system resources are the bottleneck at that time, what else is happening on the build machine, what's going on in the network (if using distributed build techniques), status/location/performance of the many caching systems involved in a build, etc.
Compiling 100 tiny C files may be faster than compiling a single huge one, or viceversa. Building small highly convoluted code can be slower than building huge amounts of straight-forward/linear code.
Even the context of the build matters - using a j factor optimized for builds on dedicated servers fine tuned for exclusive, non-overlapping builds may yield very dissapointing results when used by developers building in parallel on the same shared server (each such build may take more time than all of them combined if serialized) or on servers with different hardware configurations or virtualized.
There's also the aspect of correctness of the build specification. Very complex builds may have race conditions causing intermittent build failures with occurence rates that can vary wildly with the increase or decrease of the j factor.
I can go on and on. The point is that you have to actually evaluate your build in your very context for which you want the j factor optimized. @Jeff Schaller's comment applies: iterate until you find your best fit. Personally I'd start from the nproc value, try upwards first and downwards only if the upwards attempts show immediate degradation.
Might be a good idea to first measure several identical builds in supposedly identical contexts just to get an idea of the variability of your measurements - if too high it could jeopardise your entire optimisation effort (a 20% variability would completely eclipse a 10% improvement/degradation reading in the j factor search).
Lastly, IMHO it's better to use an (adaptive) jobserver if supported and available instead of a fixed j factor - it consistently provides a better build performance across wider ranges of contexts.
-j parameter? e.g. make -j
make -j will spawn as many jobs as the dependencies allow like a fork bomb (superuser.com/questions/927836/…); the build will crawl at best spending most CPU on managing the processes than running them (superuser.com/questions/934685/…) and in highly parallel builds the system will run out of memory/swap or pid #s and the build will fail.
Jul 3, 2015 at 11:06
If what you need is the number of cores of the processor -1, and lscpu command not found. You can use this:
make -j$(grep processor /proc/cpuinfo | tail -n 1 | awk '{print $3}')
If you'd like to write make command to use as many parallel workers as you've got virtual CPUs, I suggest to use:
nproc | xargs -I % make -j%
Which can be written as either a standalone command or as RUN directive within Dockerfile (as Docker doesn't support nested commands)
I coded a tool, called makeMax, that runs make with the optimal amount of jobs and load. Based on the best answers I found here.
lscpu | grep "^CPU(" | awk '{print $2}'
This gets you the number of your CPU cores.
Print the number of processing units available to the current process, which may be less than the number of online processors
You can use make -j$(nproc) . This command is used to build a project using the make build system with multiple jobs running in parallel.
For example, if your system has 4 CPU cores, running make -j$(nproc) would instruct make to run 4 jobs concurrently, one on each CPU core, speeding up the build process.
You can also see how many cores you have with run this command; echo $(nproc)
