When did Unix move away from storing clear text passwords in passwd? Also, when was the shadow file introduced?
For the early history of Unix password storage, read Robert Morris and Ken Thompson's Password Security: A Case History. They explain why and how early Unix systems acquired most the features that are still seen today as the important features of password storage (but done better).
- The first Unix systems stored passwords in plaintext. Unix Third Edition introduced the
cryptfunction which hashes the password. It's described as “encryption” rather than “hashing” because modern cryptographic terminology wasn't established yet and it used an encryption algorithm, albeit in an unconventional way. Rather than encrypt the password with a key, which would be trivial to undo when you have the key (which would have to be stored on the system), they use the password as the key.
- When Unix switched from an earlier cipher to the then-modern DES, it was also made slower by iterating DES multiple times. I don't know exactly when that happened: V6? V7?
- Merely hashing the password is vulnerable to multi-target attacks: hash all the most common passwords once and for all, and look in the password table for a match. Including a salt in the hashing mechanism, where each account has a unique salt, defeats this precomputation. Unix acquired a salt in Seventh Edition in 1979.
- Unix also acquired password complexity rules such as a minimum length in the 1970s.
Originally the password hash was in the publicly-readable file
/etc/passwd. Putting the hash in a separate file
/etc/shadow that only the system (and the system administrator) could access was one of the many innovations to come from Sun, dating from around SunOS 4 in the mid-1980s. It spread out gradually to other Unix variants (partly via the third party shadow suite whose descendent is still used on Linux today) and wasn't available everywhere until the mid-1990s or so.
Over the years, there have been improvements to the hashing algorithm. The biggest jump was Poul-Henning Kamp's MD5-based algorithm in 1994, which replaced the DES-based algorithm by one with a better design. It removed the limitation to 8 password characters and 2 salt characters and had increased slowness. See IEEE's Developing with open source software, Jan–Feb. 2004, p. 7–8. The SHA-2-based algorithms that are the de facto standard today are based on the same principle, but with slightly better internal design and, most importantly, a configurable slowness factor.
I don't yet have a primary source, but according to this TrustedSec post (emphasis mine):
Early systems stored passwords in plaintext but eventually this was replaced by more secure forms of password storage. Robert Morris developed crypt based on the m-209 cipher machine and it appeared in Version 3 Unix, though Crypt was not used to store passwords until 6th Edition Unix (1974).
According to multiple sources, Version 3 UNIX was released in February, 1973.
From the original paper by Thompson and Morris, we can confirm that plaintext storage was originally used:
The UNIX system was first implemented with a password file that contained the actual passwords of all the users, and for that reason the password file had to be heavily protected against being either read or written.
/etc/shadow appeared in multiple branches of UNIX as noted in other answers.
According to the History section in the wikipedia's passwd page,
Password shadowing first appeared in Unix systems with the development of SunOS in the mid-1980s, System V Release 3.2 in 1988 and BSD4.3 Reno in 1990. But, vendors who had performed ports from earlier UNIX releases did not always include the new password shadowing features in their releases, leaving users of those systems exposed to password file attacks.
System administrators may also arrange for the storage of passwords in distributed databases such as NIS and LDAP, rather than in files on each connected system. In the case of NIS, the shadow password mechanism is often still used on the NIS servers; in other distributed mechanisms the problem of access to the various user authentication components is handled by the security mechanisms of the underlying data repository.
In 1987 the author of the original Shadow Password Suite, Julie Haugh, experienced a computer break-in and wrote the initial release of the Shadow Suite containing the login, passwd and su commands. The original release, written for the SCO Xenix operating system, quickly got ported to other platforms. The Shadow Suite was ported to Linux in 1992 one year after the original announcement of the Linux project, and was included in many early distributions, and continues to be included in many current Linux distributions.