Does mmap allow creating a mapping that is much larger than the amount of physical memory?

Question

Obviously, mmap is limited by the size of the largest available block of virtual address space. But let's assume we have a 64-bit system, where this is almost unlimited for most purposes. Let's also assume no swap partition is configured.

In the case of an anonymous mapping, the size would have to be limited to the total amount of physical memory available.

However, in the case of a file-backed mapping, I wonder if there is a limit based on the amount of physical memory, e.g., the size of the mapping can be at most X times the amount of physical memory? Or will mmap allow me to map arbitrarily large files as a single chunk as long as virtual address space limits are not hit? For example, if I have 32 GiB of RAM, can I successfully mmap a 1 TiB disk file?

Asking about Linux specifically, in case it matters. (I tried with a 64 GiB file, and it seems to work. Unfortunately I don't have much more disk space available than that.)

Answer 1

Using mmap() to map a file does not copy the file to physical memory so there's no reason to limit it. As long as the entire file can be represented by the virtual address space, it can be mapped. The way this works is that each page is not resident in memory and so accessing it triggers a page fault. The kernel transparently uses this opportunity to access that part of the file. This allows a user process to operate on a very large file without doing a bunch of inefficient read(), write(), and seek() loops (although it still causes context switches, as something has to actually deal with VFS).

Note that anonymous mappings can exceed remaining physical memory if memory overcommit is enabled since the page table is set so that each of the virtual pages points to the physical zero page. Any read will return zeros, but a write will cause a page fault and the kernel will need to find a free physical page to store the data, at which time it can update the page tables. This requires you have enough physical memory+swap, of course. Overcommit is a useful optimization as most memory allocators will aggressively request memory for their heaps, even if most of it goes unused.

You can even map a file larger than the address space, but not all at once with one mmap() call.

---

As an addendum, even on a 64-bit system, you may not be able to use a full 64-bit address space:

$ grep -m1 "address sizes" /proc/cpuinfo 
address sizes   : 39 bits physical, 48 bits virtual

On that system, because the 48-bit virtual address space is split in two, you could map a file as large as 2⁴⁷ bytes or 128 TiB (actually a little less, because some address space will already be in use).