BOOTING IS HARD
Booting... well... it really is the hardest part. Every time a computer boots it basically meets itself anew. It acquaints itself with its various parts, and for each one it meets it gains capability. But it has to pull itself up by its own bootstraps, so to speak, from square one every time.
When designing a boot process the trick is to bring the machine up in stages. Your boot must be fast and reliable, and it must be both things in a completely unknown environment every time. I won't even venture into a Real/Protected mode conversation (which isn't to say that I even could), but there's a lot going on at boot. As the computer assimilates its various components each time it does so in graduated steps. Probably the most crucial of these is the move from executing on-board code to executing on-disk code, or, in other words - the kernel exec. This is when the firmware (ostensibly) surrenders to the operating system.
Many years ago this was not so much the case. It used to be the BIOS really was the Basic In/Out - regular programs would make calls to the firmware for things like drawing the screen and accessing the disk. These were called interrupts - old hats might remember them best for the thrills of delight they often found in assigning IRQs for their new dot-matrix or USR.
It is the interrupt (or
INT in assembly lingo) 13H series of functions that the BIOS offers as services for disk access. These are still even used today for BIOS systems in the boot process to make the jump from firmware to disk.
A BIOS system will check the first few bytes of each disk it finds and look for a pattern it recognizes as a Master Boot Record (or
MBR). This is a decades-old de-facto standard and includes a bit of raw, executable binary which is written to the head of the disk. The MBR marks a BIOS disk as bootable. It will stop checking when it finds one, and so practically one is all you get without some clever trickery. When it does find one it maps it to memory and executes it (in Real mode, but I'm still not going there).
The executed MBR is almost definitely not your system kernel - 512 bytes (give or take) would be pretty useless in that department. This is probably a bootloader - a program designed specifically to overcome one of the BIOS's many addressing limitations - specifically that it does not understand any kind of file system at all.
When the bootloader reads in the actual kernel and executes it in memory (as we all pray it will every time), it will probably do so by asking the BIOS via an
INT13H interrupt call. And if it does not - many fancier bootloaders will mount file systems in the conventional sense and execute the code another way - then it is very little likely that the bootloader has got so fancy without an
INT13H or two. Often bootloaders must chainload themselves - or various stages of themselves - because the 512 bytes first allocated them does not suit even their needs.
CHICKEN AND EGG
All of this is a roundabout way of discussing the disk, I know, but by this point it should be abundantly clear that the primary problem - one might call it a chicken-and-egg type - is accessing the disk that contains the program instructions about how to access disks. The key to this problem is firmware - and continues to be in very different ways even on EFI systems - and, weakest or not, the firmware is the most important link in the boot chain.
You see, once the kernel executes, and all of its myriad routines for accessing and controlling hardware initiate, all of these problems sort of disappear (or, at the least, change somewhat), because modern OSes take full-control of a system, but until they do the system's limits extend only so far as the firmware will allow. This is saying a lot - the BIOS has not changed much since the 8086. The
INT13H call is an 8086 original. Yes, there have been (myriad) extensions, and hacks of course, but
BETTER AND BETTER
Most changes to the BIOS have been mere bandages at best. It used to be a hard-disk had to be physically mapped - various and specific aspects of its geometry were referred to when data was stored to it or retrieved from it. Eventually the conventional hard disk grew to a size that prohibited this. Even just the abstract map was too much information for a BIOS to handle. As it can operate only in Real Mode, the BIOS is limited to 1 MB per memory register. Swell the cylinder map any larger than that, or make any one of its properties larger than can be addressed in so many bits, and the BIOS is literally lost - out of bounds.
This barrier has been met and broken many times. Each time the map has abstracted and encoded in some newer, clever, and less accurate way. And so these days it is literally impossible for a BIOS to accurately map a drive. Logical Block Addressing is the de-facto standard now, though some Cylinder/Head/Sector (or CHS) translations are still necessary. What the mainboard firmware has lost in accuracy/responsibility, such extensions have abstracted and added to the disk firmware responsibilities to fill the gaps.
It is this cat-and-mouse game that is referenced in your question. When the BIOS fails to understand a disk beyond a certain point due to its sheer size, then any data you might want it to retrieve for you at boot - such as a bootloader or kernel - had probably better not be located beyond that point. This is where
/boot came from.
MAYBE ACTUALLY BETTER
These days such things are, thankfully, made irrelevant by BIOS's demise. It was 30 years coming, but it has been largely replaced in the past few years by the UEFI (or EFI 2.0) standard. UEFI provides a mount from minute one, it initializes in Protected Mode, it incorporates its own bootloader, it provides reboot-persistent flash-memory variable storage, it is spec'd to handle some umpteen zetabytes or whatever per disk... and much else. It is far from perfect, but it is a vast improvement over its predecessor.
Even arguments for specialized bootloaders involving disk-encryption or layered file systems fall flat when you consider that all of these must be handled by the OS kernel anyway, and if you're provided a mount at boot, you've always got a clear-shot to executing it (especially considering that the Linux kernel, in its default configuration, is an EFI-executable all its own).
And so a separate
/boot partition should probably not overly concern you, and if you are on an EFI system, then you've probably already got an analog in the EFI-system partition anyway, as that is a requirement for booting EFI mode.
/bootpartition was explicitly enforced to be in that area to ensure that the kernel would be loadable.