Which tools for distributed builds are currently in use in popular Linux distros, like Ubuntu, Debian, RedHat, etc.?

For example, we have a huge amount of packages with dependencies and try to compile all of them for the new release of the distro. It takes a lot of time to build them one by one. I have heard about distcc, but it shares one project compilation between the servers for distributed parallel building.

Is there any tool which analyzes dependencies between the packages (creates a graph, for example) and creates a schedule for building with package-level parallelism? For example, we have packages p1, p2, p3, p4, and two servers. One server builds p1, another p2 at the same time if they are not dependent on each other.

I need examples of the projects which are being used in production.

  • 1
    Most modern distros deliver pre-compiled packages. Unless I mis-understand your question, it sounds like you are compiling most of the software for your systems locally. Is this correct? If not, what distro(s) are you using? – 0xSheepdog Apr 14 '19 at 23:24
  • 1
    I build packages for custom Debian-like distro. Sometimes I need to rebuild the whole set of packages from sources. The result is a set of .deb packages with binaries inside. But it is a long-running process, so I am interested in ways of solving this problem. – heylin_n Apr 15 '19 at 0:44
  • Sounds awesome. And challenging. Thanks for the info, I see Stephen has already answered. – 0xSheepdog Apr 15 '19 at 16:00

Each distribution tends to have its own set of tools for this, there’s not much shared software here.

Debian uses wanna-build, buildd and sbuild, which you’ll all find documented on the Debian site (follow the links too). wanna-build maintains the build queue, buildd picks a package to build, and sbuild builds it. wanna-build tracks packages with missing dependencies using the “dep-wait” state; packages can enter that state directly (if wanna-build itself can identify that dependencies are missing) or after a build fails because of missing dependencies. There’s a tutorial available if you want to set up a local build infrastructure.

Fedora uses Koji, which is extensively documented. It also involves a number of different components, including koji-hub, the centralised database front-end, and kojid , which drives the builds. I’m not as familiar with Koji though so I don’t know how it all integrates to handle build states.

Other distributions have other build systems, such as Launchpad. They all handle the same concerns you have: a centralised view of all the packages, multiple build systems, and a centralised repository which receives the build results.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.