Autoconf and Automake were set out to solve an evolutionary problem of Unix.
As Unix evolved into different directions, developers that wanted portable code tended to write code like this:
Set things up in the BSD way
Set things up in the SystemV way
As Unix was forked into different implementations (BSD, SystemV, many vendor forks, and later Linux and other Unix-like systems) it became important for developers that wanted to write portable code to write code that depended not on a particular brand of operating system, but on features exposed by the operating system. This is important because a Unix version would introduce a new feature, for example the "send" system call, and later other operating systems would adopt it. Instead of having a spaghetti of code that checked for brands and versions, developers started probing by features, so code became:
Use Send here
Use something else
Most README files to compile source code back in the 90's pointed developers to edit a config.h file and comment out that proper features available on the system, or would ship standard config.h files for each operating system configuration that had been tested.
This process was both cumbersome and error prone and this is how Autoconf came to be. You should think of Autoconf as a language made up of shell commands with special macros that was able to replace the human editing process of the config.h with a tool that probed the operating system for the functionality.
You would typically write your probing code in the file configure.ac and then run the autoconf command which would compile this file to the executable configure command that you have seen used.
So when you run
./configure && make you were probing for the features available on your system and then building the executable with the configuration that was detected.
When open source projects started using source code control systems, it made sense to check in the configure.ac file, but not the result of the compilation (configure). The autogen.sh is merely a small script that invokes the autoconf compiler with the right command arguments for you.
Automake grew also out of existing practices in the community. The GNU project standardized a regular set of targets for Makefiles:
make all would build the project
make clean would remove all compiled files from the project
make install would install the software
- things like
make dist and
make distcheck would prepare the source for distribution and verify that the result was a complete source code package
- and so on...
Building compliant makefiles became burdensome because there was a lot of boilerplate that was repeated over and over. So Automake was a new compiler that integrated with autoconf and processed "source" Makefile's (named Makefile.am) into Makefiles that could then be fed to Autoconf.
The automake/autoconf toolchain actually uses a number of other helper tools and they are augmented by other components for other specific tasks. As the complexity of running these commands in order grew, the need for a ready-to-run script was born, and this is where autogen.sh came from.
As far as I know of, Gnome was project that introduced the use of this helper script autogen.sh