4

Obviously, mmap is limited by the size of the largest available block of virtual address space. But let's assume we have a 64-bit system, where this is almost unlimited for most purposes. Let's also assume no swap partition is configured.

In the case of an anonymous mapping, the size would have to be limited to the total amount of physical memory available.

However, in the case of a file-backed mapping, I wonder if there is a limit based on the amount of physical memory, e.g., the size of the mapping can be at most X times the amount of physical memory? Or will mmap allow me to map arbitrarily large files as a single chunk as long as virtual address space limits are not hit? For example, if I have 32 GiB of RAM, can I successfully mmap a 1 TiB disk file?

Asking about Linux specifically, in case it matters. (I tried with a 64 GiB file, and it seems to work. Unfortunately I don't have much more disk space available than that.)

2

1 Answer 1

7

Using mmap() to map a file does not copy the file to physical memory so there's no reason to limit it. As long as the entire file can be represented by the virtual address space, it can be mapped. The way this works is that each page is not resident in memory and so accessing it triggers a page fault. The kernel transparently uses this opportunity to access that part of the file. This allows a user process to operate on a very large file without doing a bunch of inefficient read(), write(), and seek() loops (although it still causes context switches, as something has to actually deal with VFS).

Note that anonymous mappings can exceed remaining physical memory if memory overcommit is enabled since the page table is set so that each of the virtual pages points to the physical zero page. Any read will return zeros, but a write will cause a page fault and the kernel will need to find a free physical page to store the data, at which time it can update the page tables. This requires you have enough physical memory+swap, of course. Overcommit is a useful optimization as most memory allocators will aggressively request memory for their heaps, even if most of it goes unused.

You can even map a file larger than the address space, but not all at once with one mmap() call.


As an addendum, even on a 64-bit system, you may not be able to use a full 64-bit address space:

$ grep -m1 "address sizes" /proc/cpuinfo 
address sizes   : 39 bits physical, 48 bits virtual

On that system, because the 48-bit virtual address space is split in two, you could map a file as large as 247 bytes or 128 TiB (actually a little less, because some address space will already be in use).

2
  • Good point about anonymous mappings. I should have said that if we want to make an anonymous mapping and have all of it be usable, then its size would have to be limited by physical memory.
    – Brian Bi
    Aug 7, 2022 at 2:07
  • 1
    @BrianBi Usable for writing, at least. You can still read from an anonymous mapping larger than your physical memory because each page points to the zero page, but as soon as you try to write to a page, a fault occurs and the kernel needs to find an actual free physical page to put the data (at which point it also updates the page tables so the virtual page points to the new physical page, not the zero page).
    – forest
    Aug 7, 2022 at 2:08

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .