0

For example, would it not be beneficial to have -jN > nrcpus during the configure phase, where CPU load is low? Then have -jN = nrcpus during compilation, where load should be evenly distributed to each CPU?

How difficult would this be to implement in a build environment? Would it require changing each configuration file for each step? Is it possible to automate for multiple projects, given they use the same "make" program?

This is assuming that the workload is never I/O bound.

8
  • Related: unix.stackexchange.com/q/208568/117549 – Jeff Schaller Dec 22 '19 at 3:34
  • 2
    Not clear to me what you are asking. The typical configure script is a single threaded shell script, so there is nothing to be gained from telling make that it can run multiple jobs. To answer the question in the title, look at the -l option to make. This limits jobs being started if the load average (which is a crude approximation to CPU load if the workload is not I/O bound) is too high. – icarus Dec 22 '19 at 4:31
  • @icarus it's not clear if the OP is using configure. The configure tag was added by a moderator, and the original cmake tag removed. That being said, I don't know if cmake supports that kind of parallelism, but I know cmake supports not only make, but also other build systems like ninja. – mosvy Dec 22 '19 at 23:12
  • @mosvy cmake is a generator, which as you say can generate files such as build.ninja for ninja, Makefiles for various flavors of make, and even supports some IDEs. The typical use is a one time mkdir build ; cd build ; cmake .. which does the generation based on ../CMakeLists.txt, and then just use make as required. There is a lot more to it of course, support for cross compiling, caching build flags and rebuilding if they change etc. This invocation of cmake does a lot of the same things that a typical configure script does, e.g. finding installed optional libraries. – icarus Dec 23 '19 at 2:57
  • If this is what the OP was thinking of as the "configure" stage, then this is still single threaded. There is no -j flag to cmake. – icarus Dec 23 '19 at 3:00
1

While a specified N within make -jN is a fixed maximum, there is a way to limit parallel jobs based on system load. See make -l load or make --load-average[=load]

References:

https://www.gnu.org/software/make/manual/make.html#Options-Summary https://www.gnu.org/software/make/manual/make.html#Parallel

Note that option -l load is supported in newer gnu make, though an installed version could be older.

For selecting load, here's one example: first tune -jN for the best performance on your server (also try building sources with gcc -pipe); after observing load averages, tune -l load so the value is enough to support your build activity. Tuning -l load may also help avoid overloading the server.

1
  • Thank you. Is this only limited to gnu make? A more pertinent question with example: Assuming I am in the top lvl directory of a project and run make -j100 -l5, will the nr. of jobs be able to dynamically for that one Makefile throughout? I have only experienced using load average with multiple projects, not one project, so I have some difficulty understanding how it works. Also then for a build host, would it then be ideal to set gnu make parameters to -j(nr CPUs * 2) -l(avg. load)? That way more jobs can be created when load is not exceeded for normal operation. Assuming RAM is no issue. – avisitoritseems Jan 2 '20 at 6:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.