You should not need 32 mb, you really just need to boot up linux (the kernel) and make sure it has support for USB drives built-in.
It may be tricky to make it find the right root partition if the computer has several SATA/USB disks, but other than that, you just need to load the kernel and tell it to mount the pendrive as root.
I guess this won't make it fit nicely in a floppy (unless you master the art of shrinking kernels or own a 2.88 floppy drive and compatible floppies), but it should make the space requirement simpler.
If that fails, well, you can try finding (or coding) a small program that's able to handle USB drives and put it in a floppy. I wonder if bootloaders like Smart Boot Manager are able to chainload USB disks.
Edit: on compiling your own kernel: the procedure is not hard, it's easy. The hard part is finding what exactly do you need and what can you rule out. The Gentoo Handbook chapter on kernel configuration shows you the steps — for short, you
Get the kernel source and make sure the system has the tools to compile it (this last part is not covered by the Gentoo Handbook because, well, Gentoo always has these tools)
Configuring the kernel: if you're not used to UNIX development, many times development steps are coded in a single file with recipes, the Makefile, which is used by make. Linux uses it, and it offers several ways (interfaces) to configure Linux: make xconfig opens a graphical tool, make config does it with plain text, and I usually go for make menuconfig, that provides a nice (text) menu.
Compiling the kernel: Oh, just make, so it compiles the code and builds the image
Copying the compiled kernel to wherever it should live at
Tell the bootloader about that kernel
Now the problem is how to make something useful out of this, you want to make your kernel very small, so a possibility is to enable support for modules and just compile in-kernel whatever modular features you need during boot (includes support for the USB controller and whatever filesystem the USB drive is in), and the rest you will need should go as modules into the USB drive (they can be loaded later).
The kernel itself has targets to build a floppy disk image, make fdimage, make fdimage144 and make fdimage288, but I wonder how does this behave if it cannot build a so small image...
I also wonder if one could build a really minimalistic kernel with kexec support and no modules, that just reads the USB drive and as a binary runs something that kexecs another kernel (kexec, by analogy with the POSIX exec, replaces the running kernel)