I was reading Ritchie and Thompson's paper about the Unix file system. They write, 'It is worth noting that the system is totally self-supporting'. Were the systems before Unix not self-supporting? In what ways?
The question in your title is addressed immediately after your quote in the paper:
All Unix software is maintained on the system; likewise, this paper and all other documents in this issue were generated and formatted by the Unix editor and text formatting programs.
So “self-supporting” means that once a Unix system is setup, it is self-sufficient, and its users can use it to make changes to the system itself. “This issue” in the quote above refers to Bell System Technical Journal, Volume 57, Number 6, Part 2, July-August 1978, which was all about the Unix system (and makes fascinating reading for anyone interested in Unix and its history).
The fact that Unix is self-supporting doesn’t mean all other systems before it weren’t; but some operating systems did require the use of other systems to build them (this became more common later, in fact, with the advent of micro-computers, whose systems were often developed on minis). Unix was novel in that it also included typesetting tools, which meant that it could not only build itself, but also produce its documentation, both online and in print (I imagine Unix might not be the first such system, but this would have been at least unusual).
The PDP-11 was made by Digital Equipment Corporation, and the manufacturer's operating systems for the PDP-11 were:
Of these, at least RSTS/E required yet another operating system to generate the system. It was a single-task single-user Disk Operating System, called DOS of course, that supported little but tape, disks, a teletype, the
sysgen program that asked you configuration questions and created assembly-language configuration files, and an assembler and linker. The output of the
sysgen program was a RSTS/E bootable operating system. I can't speak for RT-11 or RSX-11 as I never had to sysgen them, but I sysgen'd RSTS/E many times in the late 1970s.
They fixed all that by RSTS/E version 6b or possibly 6c, which hosted the
sysgen program itself, via an RT-11 runtime system (and also had an RSX-11 runtime system), but all this had by then been going on since about 1968. Hence the emphasis on Unix being self-supporting.
To answer the original question Why is Unix self-supporting? it is because systems which were not self-hosting kind of sucked.
When there is another system needed to develop, the resulting system does not get the needed attention to be reliable and friendly and usable.
Early in Sequent's history, they used VAXes to develop their Unix-like operating system and other components. In relating their history, the engineers proudly state that as soon as their product was mostly reliable, the company president helped roll the VAXes away so that development had to be self hosting, and the company attributed the de-VAXing as a major milestone resulting in high reliability and driving user friendliness.
I personally experienced several annoying, non-self-hosting development systems from the 1970s: ISIS and iRMX which were reputedly developed on VAXen, though I think iRMX may have been largely self-hosting by the late 1980s.