I've got a RHEL4 machine with 1TB of memory and have discovered that malloc and mmap refuse to allocate any memory above 512GB. Meanwhile, I don't seem to have a problem with RHEL5 machines. I'm presuming, then, that it's just a matter of kernel configuration. Does anyone know what controls the maximum process size (assuming a fixed page size)?
The initial port of Linux for amd64 only supported a 40-bit virtual address space, divided in 512GB for the process and 512GB for the kernel. Current versions support 48 bits of virtual address space, divided in 128TB/128TB. This last limitation is intrinsic in the current version of the amd64 architecture: there is simply no way to configure the MMU to make more bits significant. See How does Linux support more than 512GB of virtual address range in x86-64? for an explanation of the amd64 MMU, or the AMD64 Architecture Programmer's Manual (§5.1) for the full story.
I don't know exactly when this limitation changed, but your 2.6.9 kernel is really old. Red Hat publishes a table of features and limits for RHEL 3 through RHEL 6. The maximum per-process address space on RHEL4 is 512GB. RHEL5 will get you 2TB, but you might want to upgrade directly to RHEL6 which supports the full 128TB.