17

Yesterday I was trying to compile the ROOT package from source. Since I was compiling it on a 6 core monster machine, I decided to go ahead and build using multiple cores using make -j 6. The compiling went smooth and really fast at first, but at some point make hung using 100% CPU on just one core.

I did some googling and found this post on the ROOT message boards. Since I built this computer myself, I was worried that I hadn't properly applied the heatsink and the CPU was overheating or something. Unfortunately, I don't have a fridge here at work that I can stick it in. ;-)

I installed the lm-sensors package and ran make -j 6 again, this time monitoring the CPU temperature. Although it got high (close to 60 C), it never went past the high or critical temperature.

I tried running make -j 4 but again make hung sometime during the compile, this time at a different spot.

In the end, I compiled just running make and it worked fine. My question is: Why was it hanging? Due to the fact that it stopped at two different spots, I would guess it was due to some sort of race condition, but I would think make should be clever enough to get everything in the right order since it offers the -j option.

  • 4
    It does sound like a race condition. One thing you could do is attach to the running make process (the one that is spinning) using, e.g. strace -p <pid> and see if you can find out what it's looking at/for. strace will only show you syscalls (not function calls), but it could still give you valuable info if it's spinning while looking at or for a particular file. – jlp Jul 11 '12 at 23:13
  • The thread you found via google leads to the conclusion that no one was able to compile it with -j >1. – Nils Jul 14 '12 at 20:58
  • Not related to parallel compilation, but I had a hanging makefile which took forever to debug. Turns out it was simply in the initialization of a variable, $(shell ...) was ultimately running a command which was waiting for input from stdin. This was caused when a variable was empty and no file arguments were passed to the command. – jozxyqk Dec 3 '14 at 15:50
13
+100

I don't have an answer to this precise issue, but I can try to give you a hint of what may be happening: Missing dependencies in Makefiles.

Example:

target: a.bytecode b.bytecode
    link a.bytecode b.bytecode -o target

a.bytecode: a.source
    compile a.source -o a.bytecode

b.bytecode: b.source
    compile b.source a.bytecode -o a.bytecode

If you call make target everything will compile correctly. Compilation of a.source is performed (arbitrarily, but deterministically) first. Then compilation of b.source is performed.

But if you make -j2 target both compile commands will be run in parallel. And you'll actually notice that your Makefile's dependencies are broken. The second compile assumes a.bytecode is already compiled, but it does not appear in dependencies. So an error is likely to happen. The correct dependency line for b.bytecode should be:

b.bytecode: b.source a.bytecode

To come back to your problem, if you are not lucky it's possible that a command hang in a 100% CPU loop, because of a missing dependency. That's probably what is happening here, the missing dependency couldn't be revealed by a sequential build, but it has been revealed by your parallel build.

  • Interesting. Do you know if there are any tools available that can run through a makefile and check these dependencies? – user545424 Jul 14 '12 at 19:47
  • I don't know any. In any case such a tool could only find obvious mistakes. Unless it understands the syntax for each command that appears in the Makefile, and knows what are the (potentially implicit) dependencies. – Stéphane Gimenez Jul 15 '12 at 15:23
2

I don't know how long you've had the machine, but my first recommendation would be to try a memory test and verify that the memory is functioning properly. I know it often isn't the memory that is the problem, but if it is, it is best to eliminate it as a cause first before trying to trace down other probably issues.

2

I realize this is a really old question, but it still pops up at the top of search results, so here is my solution:

GNU make has a jobserver mechanism to ensure make and its recursive children do not consume more than the specified number of cores: http://make.mad-scientist.net/papers/jobserver-implementation/

It relies on a pipe shared by all processes. Each process that wants to fork additional children has to first consume tokens from the pipe, then relinquish them when done. If a child process does not return the tokens it consumed, the top-level make while hang forever waiting for them to be returned.

https://bugzilla.redhat.com/show_bug.cgi?id=654822

I encountered this error when building binutils with GNU make on my Solaris box, where "sed" is not GNU sed. Fiddling with PATH to make sed==gsed take priority over the system sed fixed the issue. I don't know why sed was consuming tokens from the pipe, though.

0

your system might be ok, but it could be a race condition happening with make when running builds in parallel.

If something is wrong with your system, it would hang/crash for other scenarios, not just when doing parallel builds.

0

This could be a race condition, but also if all necessary compilation is done in parallel and waiting for others, linking takes your time on your machine. I think that if linking waits for previous necessary compilation in parallel, then you get high cpu frequency on linking thread whatever you compile.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.