How did the Linux Kernel project track bugs in the Early Days?

Question

We all know that Linus Torvalds created Git because of issues with Bitkeeper. What is not known (at least to me) is, how were issues/tickets/bugs tracked up until then? I tried but didn't get anything interesting. The only discussion I was able to get on the subject was this one where Linus shared concerns with about using Bugzilla.

Speculation: - The easiest way for people to track bugs in the initial phase would have been to put tickets in a branch of its own but am sure that pretty quickly that wouldn't have scaled with the noise over-taking the good bugs.

I've seen and used Bugzilla and unless you know the right 'keywords' at times you would be stumped. NOTE: I'm specifically interested in the early years (1991-1995) as to how they used to track issues.

I did look at two threads, "Kernel SCM saga", and "Trivia: When did git self-host?" but none of these made mention about bug-tracking of the kernel in the early days.

I searched around and wasn't able to get of any FOSS bug-tracking software which was there in 1991-1992. Bugzilla, Request-tracker, and others came much later, so they appear to be out.

Key questions

1. How did then Linus, the subsystem-maintainers, and users report and track bugs in those days?
2. Did they use some bug-tracking software, made a branch of bugs and manually committed questions and discussions on the bug therein (would be expensive and painful to do that) or just use e-mail.
3. Much later, Bugzilla came along (first release 1998) and that seems to be the primary way to report bugs afterwards.

Looking forward to have a clearer picture of how things were done in the older days.

Answer 1

In the beginning, if you had something to contribute (a patch or a bug report), you mailed it to Linus. This evolved into mailing it to the list (which was [email protected] before kernel.org was created).

There was no version control. From time to time, Linus put a tarball on the FTP server. This was the equivalent of a "tag". The available tools at the beginning were RCS and CVS, and Linus hates those, so everybody just mailed patches. (There is an explanation from Linus about why he didn't want to use CVS.)

There were other pre-Bitkeeper proprietary version control systems, but the decentralized, volunteer-based development of Linux made it impossible to use them. A random person who just found a bug will never send a patch if it has to go through a proprietary version control system with licenses starting in the thousands of dollars.

Bitkeeper got around both of those problems: it wasn't centralized like CVS, and while it was not Free Software, kernel contributors were allowed to use it without paying. That made it good enough for a while.

Even with today's git-based development, the mailing lists are still where the action is. When you want to contribute something, you'll prepare it with git of course, but you'll have to discuss it on the relevant mailing list before it gets merged. Bugzilla is there to look "professional" and soak up half-baked bug reports from people who don't really want to get involved.

To see some of the old bug-reporting instructions, get the historical Linux repository. (This is a git repository containing all the versions from before git existed; mostly it contains one commit per release since it was reconstructed from the tarballs). Files of interest include README, MAINTAINERS, and REPORTING-BUGS.

One of the interesting things you can find there is this from the Linux-0.99.12 README:

 - if you have problems that seem to be due to kernel bugs, please mail
   them to me ([email protected]), and possibly to any other
   relevant mailing-list or to the newsgroup.  The mailing-lists are
   useful especially for SCSI and NETworking problems, as I can't test
   either of those personally anyway.

Answer 2

The processes used news groups (USENET), and (predominantly) email. A bug "existed" as a thread, putting "[BUG REPORT]" or "LINUX BUG REPORT" in the subject was a common convention. There were no bug IDs. Given the typical user-base, a bug report often came with a patch. There was one long-forgotten software tool used: ibug (see below), other than that diff+patch.

From Linux Installation and Getting Started (Jan 1994, v2.0 archived copy) >

2.6  The Design and Philosophy of Linux

 When new users encounter Linux, they often have a few misconceptions and
 false expectations of the system. Linux is a  unique  operating  system,
 and  it is important to understand its philosophy and design in order to
 use it effectively.  Time enough for a soapbox. Even if you are an  aged
 UNIX guru, what follows is probably of interest to you.

     In  commercial UNIX development houses, the entire system is devel-
 oped with a rigorous policy of quality assurance,  source  and  revision
 control systems, documentation, and bug reporting and resolution. [...]

     With  Linux,  you  can  throw  out  the entire concept of organized
 development, source control systems, structured bug reporting,  or  sta-
 tistical  analysis.   Linux  is,  and more than likely always will be, a
 hacker's operating system.(4)

                   [...]  For the most part, the Linux community communi-
 cates via various mailing lists and USENET newsgroups. A number of  con-
 ventions have sprung up around the development effort: for example, any-
 one wishing to have their  code  included  in  the  ``official''  kernel
 should  mail it to Linus Torvalds, which he will test and include in the
 kernel [...]

1992

Here's a bug report and fix from December 1992 (0.98.6) on comp.os.linux: https://groups.google.com/d/topic/comp.os.linux/TwPA00rZMJo/discussion

Very early on there was an email list ml-linux-bugs (1992/1993), from this early FAQ in the Slackware 1.01 distribution:

VI.01) It seems that $#@! ported on linux don't run correctly, what do I do about reporting bugs?

[...] Note that my "[email protected]" bug reporting list has been phased out. It turns out that Linux has so few bugs, most of which are resolved on the newsgroup or through Linus before I can accumulate them and post. :) In short: if there's a bug in Linux or in Linux-ported software, it will usually be fixed in the next patchlevel or version.

There was the "linux-kernel" email list (which ran on the original vger), newsgroups alt.os.linux, then comp.os.linux (which quickly split to a hierarchy in 1993).

This early Linux FAQ (v1.11 Nov 1992) from comp.os.linux also suggests emailing Linus directly.

In 1992 Matt Welsh (Running Linux, Linux Bible, TLDP) announced ibug to assist in generating emailed bug reports (ironically, you could not run this on Linux at that time since it lacked sufficient networking to be able to send an email).

An email bug report template linux.temp was periodically posted on comp.os.linux too, and updates to a bug report had an update template linux.fix.temp.

There was also a patch repository (FTP), as far as I can tell this was mostly (not exclusively) for patches to programs for porting to Linux.

1993-1994

CVS copies of the kernel source were common, the earliest I can find is Dirk Steinberg's, from kernal-0.99.14 era. The first announcement I can find is from Jan 1993 on linux-activists. You can still find archived copies (1994). Dirk also maintained cvs binaries and libc source in CVS.

CVS wasn't used to track bugs in the contemporary sense, some developers preferred to use it, and patches were frequently submitted in the form of cvs generated diffs.

1995-1996

Around this time (Oct 1995) David S. Miller started using CVS for the SPARC port of the Linux kernel (The Linux/SPARC port). By Feb 1996 several other kernel developers were independently using CVS to keep track of patches, from linux-kernel this thread and this thread: Alan Cox, Stephen Tweedie, Kai Henningsen. (The second thread reports Russ Nelson stating first-hand Linus' aversion to CVS.)

1997-1998

In April 1998, shortly after the birth of Linus' second child the issue of CVS came up again, from linux-kernel see this subthread (Linus reiterates his concerns about CVS there directly).

In Dec 1997, Andrew Tridgell released jitterbug, a web-based bug tracker. By June 1998 the "linux-patches" JitterBug was being advocated on linux-kernel by Alan Cox. This was as far as I can tell, the first actual bug tracking system used by Linus and other key developers, sadly the "linux-patches" instance is no longer online.

In September 1998, bitkeeper is first promoted on linux-kernel by Larry McEvoy.

1999 and later

By 1999/2000 the lkml FAQ started referring (Q 1-16) to the CVS tree on (the original) vger. This was maintained at the time by Andrew Tridgell.

By December 2001, Jitterbug had fallen out of favour, see this linux-kernel thread, Linus, Alan Cox and many others get involved in discussing why.

By Jan 2002, Linus started getting interested in bitkeeper (already used by the PowerPC Linux kernel team).

In Feb 2002 Linus started using Bitkeeper for the 2.5 development tree.

In Nov 2002 the OSDL hosted Linux Bugzilla for the 2.5 tree was announced. (If you haven't already read the bugzilla link in the question, go and read it now, it contains vintage Linus rants).

In April 2005 Linus announced a move away from BitKeeper, around the time he first mentioned git by name. Shortly after git had become capable of self-hosting, Linus ceased using BitKeeper and started using git for the kernel.

In December 2008 the Patchwork patch tracker for linux-kernel was announced, this is a SCCS-agnostic web-based patch tracker that integrates with mailing lists to track patches and followups. Its use continues to this day, there are approximately 40 lists tracked this way on https://patchwork.kernel.org/ , though not all are active.

References

Useful references:

• Essence of distributed work: The case of the Linux kernel (Jae Yun Moon, Lee Sproull) http://www.firstmonday.org/ojs/index.php/fm/article/view/801/710 (Nov 2000)
• Guidelines for reporting Linux bugs (Jul 1992) http://www.linuxmisc.com/19-linux/c27174dbc2bf7185.htm
• Kenneth R. Saborio's archive of important Linux postings/emails: http://www.informatica.co.cr/linux/index.htm (1991-2005)
• linux-kernel archives from today (Nov 2014) back to 1995 http://lkml.iu.edu/hypermail/linux/kernel/ (sadly, the first email is from June 1995 where the administrator Chris Dent announces he has lost the earlier archives ...) LKML archive only goes back to 1996
• Fragments from linux-devel 1993-1994 from tsx11 http://www.ibiblio.org/pub/historic-linux/ftp-archives/tsx-11.mit.edu/Oct-07-1996/mail-archive/linux-devel/ (disregard the dates in the URL and on the files)
• Version Management Tools: CVS to BK in the Linux Kernel, Sjaikh & Cornford (ca. 2003)
• See also this Hacker News thread (March 2015) as the 10th anniversary of git approaches: https://news.ycombinator.com/item?id=9263336

Answer 3

I can tell how bug reporting is handled for the development of git itself.

They do not use any bug tracking software. Bugs are reported and discussed on the development mailinglist. It is possibly surprising, but works very well.

The question or proposal to use some bug tracking software comes up often, so you can learn a lot about this from searching the git mailing lists archives.

It's not about "we did not yet find a bug tracker that's good enough";
But it's also not about "we have a superior method" either.

With this method, the maintainer of the project or sub-project - something like a lead developer - has an important role as informal moderator of the development list.
Handling bugs is part of it, and it does not seem to be a trivial task to manage bugs this way; It certainly depends on the skills of persons in that role.

The most formal part of the method is a weekly status summary message.
It lists things currently going on on the various branches as short items. For an example from the git development, see this at the newsgroup gmane.comp.version-control.git mirroring the mailing list: What's cooking in git.git

What I can say for sure: If you have a maintainer who is good at this, it works extremely well.
For example, I'd be very surprised if the introduction of a bug tracker had a positive effect productivity on implemented features and quality, even in the long term after amortization of the overhead of change.

For the Linux kernel, it's similar like how it's done for git, up to today.
The development mailing lists for Linux kernel development are certainly important. But it's not one list as a central place as with git. There are separate lists for subtopics, like filesystems, or networking.
Because there are separate topics, handled by mostly separate developers, it's possible that some groups do use tools locally to their group.