Tell me more ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.
up vote 5 down vote favorite
1

I am doing a build on a Linux machine with ubuntu 10.04 on it. I want to know how can I really speed up my build. I have 4 CPUs and lots of RAM. I already reniced the process group to -20. Is there something else I can do?

share|improve this question
No mention of what your doing or in what area you are looking for speed? There are more trade off decisions to be made than magic "go faster" settings. – Caleb Aug 25 '11 at 20:07
I meant to reduce the build time by leveraging some Linux settings. – abc Aug 25 '11 at 20:16
2  
What are you building? – Caleb Aug 25 '11 at 20:19

3 Answers

up vote 13 down vote accepted

Most software build processes use make. Make sure you make make use the -j argument with a number usually about twice the number of CPUs you have, so make -j 8 would be appropriate for your case.

share|improve this answer
1  
Double the number of cores? I always heard number of cores plus one – Michael Mrozek♦ Aug 25 '11 at 20:25
1  
I think that depends on whether your jobs are large or small. +1 makes sense for big jobs, x2 for lots of little ones. My testing with building the kernel on my 3 core machine shows that x2 (-j 6) is just about the sweet spot. More actually goes a little faster but the machine becomes unacceptably unresponsive. I like to be able to still type while something is going on :) – Caleb Aug 25 '11 at 20:28
4  
Usually the "optimal" number of processes is between n+1 and 2n+1 depending on how much time is spent in I/O activity. The idea is to have some process ready to run each time one of the running processes needs to stop for some I/O (or ends). If you processes are CPU bound go for n+1. If you processes are I/O bound go for 2n or even 3n (if your disks are fast enough). In my experience, 2n is a good value to compile something. – andcoz Aug 25 '11 at 21:25
1  
Beware, this doesn't work with all possible make-based build systems. In particular, recursive builds where a top-level Makefile kicks off make(1) in subdirectories in a specific order will generally fail to build correctly when you let make(1) build in parallel unless you are very careful about documenting dependencies in the top-level Makefile. While it can be made to work, parallel builds are the best excuse I know to rethink your build system to avoid the need for recursive builds. – Warren Young Aug 26 '11 at 2:59
@Warren Young, yes we have also seen problems with parallel builds. We have a recursive build environment as you have described above. – abc Aug 26 '11 at 5:16
show 1 more comment
up vote 10 down vote

Reniceing the process group to -20 is a bad idea. This niceness level should be used only by the top-priority system-critical tasks. Otherwise you can loose responsiveness or even freeze the system. And the potential compilation-time benefit would be marginal.

Apart from what Caleb already suggested, if you compile a lot, you can also speed up builds using compiler cache or distribute the build tasks across multiple computers across a network using distcc or icecream.

share|improve this answer
up vote 1 down vote

Adding to rozcietrzewiacz's answer:

First big improvement would be the disks, use faster IOPs disks, i.e. something that runs at 10,000 or 15,000 rpm, ideally SSD and then you are only limited by your purse.

Random example, Samsung's 6Gbps SATA 2 SSD:

http://www.engadget.com/2011/08/17/samsungs-6gbps-ssd-gets-a-consumer-label-october-ship-date/

OCZ have some of the most extreme hardware, 500,000 IOPs:

http://www.engadget.com/2011/08/02/oczs-z-drive-r4-pcie-ssd-offers-2-800mb-sec-500-000-iops-pl/

You could try copying the tree to a tmpfs file system but Linux's file cache is already pretty good.

For rapid development usage the build would go faster if you disabled all the optimisations. These tend to be '90s era tweaks when processors were slow, it's pretty much all about IO now.

The most convoluted approach I can imagine is using cramfs to store a large part of the tree that doesn't change. Using a compressed file system would mean less disk accesses underneath.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.