What is the source of the "compile it yourself" mentality in linux [closed]

Question

I used Linux a bit in college, and am familiar with the terms. I develop in .NET languages regularly, so I'm not computer illiterate.

That said, I can't really say I understand the "compile it yourself" [CIY] mentality that exists in *nix circles. I know it's going away, but still hear it from time to time. As a developer, I know that setting up compilers and necessary dependencies is a pain in the butt, so I feel like CIY work flows have helped to make *nix a lot less accessible.

What social or technical factors led to the rise of the CIY mentality?

Answer 1

Very simply, for much of the history of *nix, there was no other choice. Programs were distributed as source tarballs and the only way you had of using them was to compile from source. So it isn't so much a mentality as a necessary evil.

That said, there are very good reasons to compile stuff yourself since they will then be compiled specifically for your hardware, you can choose what options to enable or not and you can therefore end up with a fine tuned executable, just the way you like it. That, however, is obviously only something that makes sense for expert users and not for people who just want a working machine to read their emails on.

Now, in the Linux world, the main distributions have all moved away from this many years ago. You very, very rarely need to compile anything yourself these days unless you are using a distribution that is specifically designed for people who like to do this like Gentoo. For the vast majority of distributions, however, your average user will never need to compile anything since pretty much everything they'll ever need is present and compiled in their distribution's repositories.

So this CIY mentality as you call it has essentially disappeared. It may well still be alive and kicking in the UNIX world, I have no experience there, but in Linux, if you're using a popular distribution with a decent repository, you will almost never need to compile anything yourself.

Answer 2

There are a few causes for that mentality, from end users, distribution maintainers, and code suppliers/developers/project groups, and each and every one of them is perfectly valid.

The Open Source aspect

There are some who enjoy knowing they are using Free software, and validate that by choosing to compile from source. This is where things like the Linux From Scratch project/howto/guide/book come in.

The optimization and options aspect

Want to compile stuff with specific optimizations for your particular CPU architecture? Perhaps there is a compile time option (or patch to create one) to enable or disable a particular feature that you need. Examples of this could be patching postfix to have the ability to manage quotas, or using a distribution like Gentoo where you can opt to not use systemd, or you opt specifically to support ogg/theora/vorbis/whatever and NOT mp3 due to licensing issues or whatever.

The CPU Architecture aspect

Does your workplace use high end non-x86/amd64 machines? The package you need/want may not be available pre-compiled for your CPU architecture, much less whatever distribution you are running. Granted, most places running this kind of hardware are also on support from IBM, etc. and don't go installing/compiling stuff willy-nilly. But what if you pick up one from a surplus sale, dig out an old iMac w/PPC processor, etc?

The Distribution aspect

Distribution "families" - ie, Debian w/ Ubuntu, Mint, et al and RedHat with CentOS, Whitebox, Fedora, et al - all use different package formats. And each version ships with different library versions, etc. Even for a simple single file shell script setting up a proper Debian .deb file takes time and effort. If you wrote some software to scratch some itch, and wanted to make it Free and post it on gitlab, your own webserver, whatever, would you rather just post a generic .tar.gz file of source with instructions on building or would you rather package up versions for 2 versions of Debian (stable and testing, maybe oldstable), multiple versions of Redhat and Fedora as RPMs, a TGZ for Slackware, an ebuild profile for Gentoo, etc. etc. etc.

Answer 3

As @terdon says, nowadays the need to compile things are pretty much slim, especially for home users.

In the past, in the Unix world, I was highly dependent on compiling sources, for instance, as I was managing Solaris, AIX, Ultrix, Digital Ultrix and HP/UX systems which sometimes were no longer maintained by the vendor, or which implementations of common services were far behind what was commonly used by others Unixes, including Linux.

There are still genuine needs for compiling things in the present, either to get some more obscure or obsolete piece of software that is not in the repositories, or use a more recent version of a package for which you do not have compatible binaries, or when you want to add extra functionality or rarely, if you are able to write a patch or module for it.

I also had to compile software by hand when doing reengineering of systems for porting to Debian and/or new versions of Debian that had a framework which was no longer supported by the OS.

For instance, in the past I had to compile by hand DHCP daemons to have the support for (by then recent) Windows changes to the protocol, or to support specific patches for provisioning in the Telecom world.

I still keep in my local repository debs for FreeRadius versions compiled by myself from the dev git repo, as there were a string of stable versions that had (serious) bugs in Debian, and usually the corresponding .debs for Debian/Ubuntu have not been adequate for our needs.

And it goes without saying that often in a while we also have to run/or compile stuff written by ourselves.

Installing the dependencies nowadays is not as hard as in the past, and some software even have customised rule files for some common Linux distributions that name the dependencies to compile and do the heavy work of creating the package file with the list of dependencies built in. Installing such a package from a local repository is not much different from installing the same package from the official repositories.

Answer 4

What social or technical factors led to the rise of the CIY mentality?

The root cause is pretty obviously the technical reason: Binary-portability is harder than source-portability. Outside of distro packages, most Free software is still only available in source form because that's vastly more convenient for the author(s)/maintainer(s).

Until Linux distros started packaging most things that average people would want to use, your only option was to get the source and compile it for your own system. Commercial Unix vendors typically didn't include stuff that almost everyone wanted (e.g. a nice shell like GNU bash or similar), just their own implementation of sh and/or csh, so you needed to build stuff yourself if you (as a sys-admin) wanted to provide a nice Unix environment to your users for interactive use.

The situation now, with most people being the only admin and only user of the machine sitting on their desktop, is vastly different from the traditional Unix model. A sysadmin maintained the software on the central system, and on everyone's desktop. (Often by having people's workstations just NFS-mount /opt and /usr/local/ from the central server, and installing stuff there.)

---

Before things like .NET and Java, true binary-portability across different CPU architectures was impossible. Unix culture evolved with source-portability as the default for this reason, with little effort to even try to enable binary-portability until recent Linux efforts like LSB. For example, POSIX (the major Unix standard) only attempts to standardize source-portability, even in recent versions.

Related cultural factor: Early commercial AT&T Unix came with source code (on tapes). You didn't have to build the system from source, it was just there in case you wanted to see how something really worked when the docs weren't enough.

Wikipedia says:

"The Unix policy of extensive on-line documentation and (for many years) ready access to all system source code raised programmer expectations, and contributed to the 1983 launch of the free software movement."

I'm not sure what motivated this decision, since giving customers access to the source code of commercial software is unheard-of these days. There are clearly some early cultural biases in this direction, but perhaps that grew out of Unix's roots as a portable OS written mostly in C (not assembly language) that could be compiled for different hardware. I think many earlier OSes had more of their code written in asm for a specific CPU, so source-level portability was one of early Unix's strengths. (I may be wrong about this; I'm not an expert on early Unix, but Unix and C are related.)

---

Distribution of software in source form is by far the easiest way to let people adapt it to whatever system they want it to run on. (Either end-users or people packaging it for a Linux distro). If software has already been packaged by/for a distribution, end-users can just use that.

But it's far too much to expect authors of most packages to make binaries for every possible system themselves. Some major projects provide binaries for a few common cases (especially x86/windows where the OS doesn't come with a build environment, and the OS vendor has put a major emphasis on distribution of binary-only installers).

Getting a piece of software to run on a different system from the one the author used might even require some small changes, which are easy with source. A small one-off program that someone wrote to scratch their own itch probably has never been tested on most obscure systems. Having the source makes it possible to make such changes. The original author might have overlooked something, or intentionally wrote less portable code because it saved a lot of time. Even major packages like Info-ZIP didn't have testers on every platform right away, and needed people to send in their portability patches as problems were discovered.

(There are other kinds of source-level portability issues that only happen because of differences in build env, and aren't really relevant to the issue here. With Java-style binary portability, auto-tools (autoconf/auto-make) and similar things like cmake wouldn't be needed. And we wouldn't have things like some systems require the inclusion of <netinet/in.h> instead of <arpa/inet.h> for ntohl(3). (And maybe we wouldn't have ntohl() or any other byte-order stuff in the first place!)

---

I develop in .NET languages regularly, so I'm not computer illiterate.

Compile-once, run-anywhere is one of the major goals of .NET and also Java, so it's fair to say that entire languages have been invented in an effort to solve this problem, and your dev experience is with one of them. With .NET, your binary runs on a portable runtime environment (CLR). Java calls it's runtime environment the Java Virtual Machine. You only need to distribute one binary that will work on any system (at least, any system where someone has already implemented a JVM or CLR). You can still have portability problems like, / vs \ path separators, or how to print, or GUI layout details, of course.

A lot of software is written in languages that are fully compiled into native code. There's no .net or java bytecode, just native machine-code for the CPU it will run on, stored in a non-portable executable file format. C and C++ are the major examples of this, especially in the Unix world. Obviously this means a binary has to be compiled for a specific CPU architecture.

Library versions are another problem. Libraries can and often do keep the source-level API stable while changing the binary-level ABI. (See Difference between API and ABI.) For example, adding another member to an opaque struct still changes its size, and requires a recompile with headers for the new library version for any code that allocates space for such a struct, whether it's dynamic (malloc), static (global), or automatic (local on the stack).

Operating systems are also important. A different flavour of Unix for the same CPU architecture might have different binary file formats, a different ABI for making system calls, and different numeric values for constants like fopen(3)'s O_RDONLY, O_APPEND, O_TRUNC.

Note that even a dynamically-linked binary still has some OS-specific startup code that runs before main(). On Windows, this is crt0. Unix and Linux have the same thing, where some C-Runtime Startup code is statically linked into every binary. I guess in theory you could design a system where that code was dynamically linked too, and part of libc or the dynamic linker itself, but this is not how things work in practice on any OS I'm aware of. That would only solve the system-call ABI problem, not the problem of numeric values for constants for standard-library functions. (Normally system-calls are made through libc wrapper functions: A normal x86-64 Linux binary for source that uses mmap() won't include the syscall instruction, just a call instruction to the libc wrapper function of the same name.

This is part of why you can't just run i386-FreeBSD binaries on i386-Linux. (For a while, the Linux kernel had a system-call compatibility layer. I think at least one of the BSDs can run Linux binaries, with a similar compat layer, but you of course need Linux libraries.)

---

If you wanted to distribute binaries, you'd need to make a separate one for every combination of CPU/OS-flavour+version/installed-library-versions.

Back in the '80s / '90s, there were many different types of CPU in common use for Unix systems (MIPS, SPARC, POWER, PA-RISC, m68k, etc.), and many different flavours of Unix (IRIX, SunOS, Solaris, AIX, HP-UX, BSD, etc.).
And that's just Unix systems. Many source packages would also compile and work on other systems, like VAX/VMS, MacOS(m68k and PPC), Amiga, PC/MS-DOS, Atari ST, etc.

There are still many CPU architectures and OSes, although now a large majority of desktops are x86 running one of three major OSes.

So there's already more CPU/OS combinations than you can shake a stick at, even before you start thinking about dependencies on 3rd-party libraries that might be at different versions on different systems. (Anything that's not packaged by the OS vendor would have to be installed by hand.)

Any paths that are compiled into the binary are also system-specific. (This saves RAM and time compared to reading them from a config file at startup). Old-school Unix systems typically had a lot of hand-customized stuff, so there's no way you could make any valid assumptions about what's where.

Distributing binaries was totally infeasible for old-school Unix except for major commercial projects that can afford to build and test on all the major combinations.

Even making binaries for just i386-linux-gnu and amd64-linux-gnu is hard. Much time and effort has been spent on things like the Linux Standard Base to make portable binaries possible. Even statically linking binaries doesn't solve everything. (e.g. how should a word-processing program print on a RedHat system vs. a Debian system? How should the install add a user or group for a daemon, and arrange for its startup script to run after every reboot?) Those are not great examples, because recompiling from source doesn't solve them.

---

Besides all that, back in the day memory was more precious than it is now. Leaving out optional features at compile-time can create smaller binaries (less code size) that also use less memory for their data structures. If a feature required an extra member in every instance of a certain class or struct to track something, disabling that feature will shrink the object by 4 bytes (for example), which is nice if it's an object that the program allocates 100k of.

Optional compile-time features these days are most often used to make extra libraries optional. e.g. you can compile ffmpeg with or without libx264, libx265, libvorbis, and many other libraries for specific video/audio encoders, subtitle handling, etc. etc. More commonly, a lot of things can be compiled with or without libreadline: if it's available when you run ./configure, the resulting binary will depend on the library, and provide fancy line-editing when reading from a terminal. If it's not, then the program will use some fall-back support to just read lines from the stdin with fgets() or something.)

Some projects do still use optional features to leave out un-needed code for performance reasons. e.g. the Linux kernel itself can be built without SMP support (e.g. for an embedded system or an ancient desktop), in which case a lot of the locking is simpler. Or with many other optional features that affect some of the core code, not just leaving out drivers or other hardware features. (Although arch-specific and hardware-specific config options account for a lot of the total source code. See Why is the Linux kernel 15+ million lines of code?)