How does linux on x86_64 recover the 'lost' physical memory masked by memory-mapped IO?

Question

/proc/iomem indicates that significant address space is mapped to PCI devices, such as a video card on my box: e0000000-efffffff : 0000:01:00.0 which is 250MB if my math is correct. On a 64 bit desktop with only 16GB RAM I assume there is some trick that linux or all modern kernels can do to recover that part of physical memory, but how exactly?

A somewhat related question - if northbridge/memory controller routes memory/io accesses based on some programmable rule, such that for write access to memory mapped regions(for example, to pci devices), RAM does not even know about those writes since they are routed away, then there should be some sort of 'routing table'? And where does such a table live? How does linux kernel access this table?

Answer 1

Just found that there are a couple wiki pages related to this topic: PCI_hole and 3_GB_barrier

Nowadays on x86, PCI hole can be dealt with via memory remapping, but this does not recover ALL RAM addresses stolen by MMIO, such as multiple small regions below 16M - chipset only has capabilities to remap limited number of regions.

Answer 2

As you have a 64-bit operating system, you could enable the BIOS setting "Above 4G Decoding", "64-bit I/O address decoding" or whatever it's called by your system/motherboard vendor. With that setting enabled, any MMIO hardware that is capable of dealing with 64-bit addresses gets mapped to addresses outside the traditional 32-bit range, minimizing the conflicts with memory and so reducing the need for remapping slots.

On my system, the resulting mapping for GPU looks like this:

6000000000-600fffffff : 0000:01:00.0

Also, 250MB is just about 1.5% of 16 GB; if getting the last 1.5% of memory is truly critical, you might get a noticeable performance benefit from getting more RAM if at all possible. Just saying...

As far as I know, the "routing table" for memory remapping is at least partly implemented in chipset hardware, and is very chipset-specific, so it is normally set up at boot time by the system firmware. If any run-time access is possible, I would expect it to be through the ACPI firmware routines; otherwise the kernel would have to have specific routines for each chipset.

(Yes, the kernel has hardware-model-specific quirk routines to work around known hardware bugs; but getting deeper than that and bypassing the ACPI abstraction provided by the system firmware would require a great deal more effort, something like coreboot.)