22

I just read here:

  • up to 128TiB virtual address space per process (instead of 2GiB)
  • 64TiB physical memory support instead of 4GiB (or 64GiB with the PAE extension)

Why is that? I mean, the physical memory support is being limited by the kernel or by the current hardware?

Why would you need twice the virtual memory space than the physical memory you can actually address?

  • You can also add swap. – Thorbjørn Ravn Andersen Aug 17 '17 at 12:19
  • 2
    that's a lot of RAM... – dalearn Aug 17 '17 at 17:56
  • 4
    @dalearn - you know, when I first learned that you could get bank-switched memory expansion for 8-bit micros that let them have up to 4096KB, I said exactly the same thing... – Jules Aug 17 '17 at 21:57
34

Those limits don't come from Debian or from Linux, they come from the hardware. Different architectures (processor and memory bus) have different limitations.

On current x86-64 PC processors, the MMU allows 48 bits of virtual address space. That means that the address space is limited to 256TB. With one bit to distinguish kernel addresses from userland addresses, that leaves 128TB for a process's address space.

On current x86-64 processors, physical addresses can use up to 48 bits, which means you can have up to 256TB. The limit has progressively risen since the amd64 architecture was introduced (from 40 bits if I recall correctly). Each bit of address space costs some wiring and decoding logic (which makes the processor more expensive, slower and hotter), so hardware manufacturers have an incentive to keep the size down.

Linux only allows physical addresses to go up to 2^46 (so you can only have up to 64TB) because it allows the physical memory to be entirely mapped in kernel space. Remember that there are 48 bits of address space; one bit for kernel/user leaves 47 bits for the kernel address space. Half of that at most addresses physical memory directly, and the other half allows the kernel to map whatever it needs. (Linux can cope with physical memory that can't be mapped in full at the same time, but that introduces additional complexity, so it's only done on platforms where it's require, such as x86-32 with PAE and armv7 with LPAE.)

It's useful for virtual memory to be larger than physical memory for several reasons:

  • It lets the kernel map the whole physical memory, and have space left for virtual mappins.
  • In addition to mappings of physical memory, there are mappings of swap, of files and of device drivers.
  • It's useful to have unmapped memory in places: guard pages to catch buffer overflows, large unmapped zones due to ASLR, etc.
  • 9
    The 46-bit limitation on physical memory is related to the Linux memory map: it includes a full mapping of physical memory in kernel space, which means physical memory can only correspond to a quarter of the available address space. – Stephen Kitt Aug 17 '17 at 8:26
  • Could anybody elaborate on the @StephenKitt comment? I'm very interested in understanding that but even after read the reference he cited I don't get it ;) – gsi-frank Aug 17 '17 at 20:48
  • @gsi-frank It's convenient for the kernel to have the whole physical memory mapped permanently. So in a 2^48 address space, 2^47 goes to userland addresses, 2^46 goes to kernel addresses and 2^46 is for physical memory addressing. – Gilles 'SO- stop being evil' Aug 17 '17 at 21:10
  • @gsi-frank If you can get hold of a copy of the classic book "Developing your own 32-bit Operating System", it goes into substantial depth about the reason the author made a similar decision for his own OS (in that case, dividing the 4GiB virtual address space of the 80386 into a 2GiB kernel segment that contains a 1GiB physical RAM mapping and a 2GiB user segment). Anyone interested in OS internals should probably read it -- it provides a full design of a simple enough to understand but advanced enough to be useful OS kernel. – Jules Aug 17 '17 at 22:05
  • Since version 4.13 of the kernel, x86-64 (and some other architectures) can be built with five-level pagetables, which increase the address space on x86-64 to 52 bits for physical RAM, and 57 bits for virtual (4 PiB / 128 PiB). Note that the memory map in kernel space introduces security issues so that’s liable to change in the near-ish future. – Stephen Kitt Sep 29 '17 at 13:31
9

I don't know why, but I can think of seven reasons why it'd be useful to support twice as much address space as physical memory.

  1. The first is so that you can run applications that need the extra memory -- even if it means swapping to disk.
  2. Cleaner memory layouts to partition memory usage. E.g., an OS might take higher-numbered addresses and leave lower-numbered addresses for applications to make separation cleaner.
  3. Address space layout randomization is a bit more effective.
  4. Marking pages as executable may mean leftover memory.
  5. Memory-mapped I/O.
  6. Memory allocation is easier: one can allocate bigger chunks at a time.
  7. Reduced memory fragmentation
  • 1
    Thanks! 1) is so evident and basic that I feel embarrassed for the question ;) – gsi-frank Aug 16 '17 at 20:51
  • 2
    (3) is really important too. You really want a virtual address space that's orders of magnitude larger than the amount of memory you'll be allocating so that random guesses almost surely result in traps. – R.. Aug 17 '17 at 16:50
6

Those are hardware limitations. Current x86_64/amd64 hardware allows 48-bit virtual addresses and various size (depends on the implementation—e.g, my workstation here only supports 36 bits) physical addresses. The Linux kernel splits virtual address space in half (using half for the kernel, half for userspace—just like it does on x86).

So you get:

2⁴⁸ bytes ÷ 2 = 2⁴⁷ bytes = 128 TiB

Physical address size is often smaller because it's actually physical. It takes up pins/pads, transistors, connections, etc., on/in the CPU and trace lines on the board. Probably also the same in the chipsets. It makes no sense to support an amount of ram that is inconceivable over the processor core's or socket's design lifespan—all those things cost money. (Even with 32 DIMM slots and 64GiB DIMMs in each, you're still only at 2TiB. Even if DIMM capacity doubles yearly, we're 5 years away from 64TiB.

As Peter Cordes points out, people are now attaching non-volatile storage such as 3D XPoint to the memory bus, which makes running out of address space conceivable. Newer processors have extended the physical address space to 48 bits; it's possible the Debian wiki just hasn't been updated.

  • Non-volatile storage connected directly to the memory bus (e.g. 3D XPoint) is becoming a thing, and this could greatly increase demand for physical address space in the next few years (since it's denser than DRAM, and it's useful to have boatloads of it in more cases than it's useful to have boatloads of RAM). See zdnet.com/article/the-non-volatile-memory-revolution for a not-very-technical article (or google for better stuff). Intel Skylake supports it with its clflush and clflushopt instructions. – Peter Cordes Aug 17 '17 at 10:42
  • 1
    You can already buy systems with up to 12TiB of RAM in 96 slots (Tyan’s four-socket HPC system for example), so 64TiB might be less than five years away. And some people do buy them and fit them with that much RAM... – Stephen Kitt Aug 18 '17 at 8:42
  • @StephenKitt hmm, it's OK because DIMM capacity takes closer to 3 years to double 😁 – derobert Aug 18 '17 at 8:51
  • It turns out you can actually buy systems with 64 TiB of RAM right now. – Stephen Kitt Sep 29 '17 at 13:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.