3

Operating System Concepts says

Consider a sequential read of a file on disk using the standard system calls open(), read(), and write(). Each file access requires a system call and disk access.

Alternatively, we can use the virtual memory techniques discussed so far to treat file I/O as routine memory accesses. This approach, known as memory mapping a file, allows a part of the virtual address space to be logically associated with the file. As we shall see, this can lead to significant performance increases. Memory mapping a file is accomplished by mapping a disk block to a page (or pages) in memory. Initial access to the file proceeds through ordinary demand paging, resulting in a page fault. However, a page-sized portion of the file is read from the file system into a physical page (some systems may opt to read in more than a page-sized chunk of memory at a time). Subsequent reads and writes to the file are handled as routine memory accesses. Manipulating files through memory rather than incurring the overhead of using the read() and write() system calls simplifies and speeds up file access and usage.

Could you analyze the performance of memory mapped file?

If I am correct, memory mapping file works as following. It takes a system call to create a memory mapping. Then when it accesses the mapped memory, page faults happen. Page faults also have overhead.

How does memory mapping a file have significant performance increases over the standard I/O system calls?

Thanks.

8

Memory mapping a file directly avoids copying buffers which happen with read() and write() calls. Calls to read() and write() include a pointer to buffer in process' address space where the data is stored. Kernel has to copy the data to/from those locations. Using mmap() maps the file to process' address space, so the process can address the file directly and no copies are required.

There is also no system call overhead when accessing memory mapped file after the initial call.

If you are interested in the practical details, there is related question on Stack Overflow: mmap() vs. reading blocks

  • Thanks. What is "copying buffers which happen with read() and write() calls"? Why does memory mapping a file not have it? – Tim Oct 12 '18 at 2:53
  • "Calls to read() and write() include a pointer to buffer in process' address space where the data is stored. " Memory-mapped file doesn't have such a buffer, but instead has a memory region mapped to a file. So are they not equivalent? – Tim Oct 12 '18 at 18:45
  • 2
    Not exactly. The kernel may cache the file, so read()/write() calls copy from the cache to process' buffers. With mmap() process can access those (cache) pages directly. – sebasth Oct 13 '18 at 10:29
  • Thanks. invoid *mmap(void *addr, size_t len, int prot, int flag, int fd, off_t off );, the addr argument lets us specify the address where we want the mapped region to start. (1) "Calls to read() and write() include a pointer to buffer in process' address space where the data is stored. " Isn't addr argument of mmap() "a pointer to buffer in process' address space where the data is stored"? – Tim Oct 14 '18 at 4:02
  • 1
    (2)" mmap() process can access those (cache) pages directly". Doesn't mmap() access the mapped region pointed by addr argument of mmap() "? Isn't the mapped region part of the process address space? – Tim Oct 14 '18 at 4:04
5

First, in most IO operations the characteristics of the underlying storage hardware dominates the performance. A poorly-configured RAID5 array of twenty-nine S-L-O-W 5400 rpm SATA disks on a slow, memory-starved system using S/W RAID with mismatched block sizes and misaligned file systems is going to give you poor performance compared to a properly configured and aligned SSD RAID 1+0 on a high-performance controller despite any software tuning you might try.

But the only way mmap() can be significantly faster is if you read the same data more than once and the data you read doesn't get paged out between reads because of memory pressure.

Memory map steps:

  1. System call to create virtual mappings - very expensive
  2. Process accesses memory for the first time, causing a page fault - expensive (and may need to be repeated if paged out)
  3. Process actually reads the memory

If the process only does steps 2 and 3 once for each bit of data read, or the data gets dropped from memory because of memory pressure, mmap() is going to be slower.

read() steps:

  1. System call copies data from disk to page cache (may or may not page fault, data may already be in page cache causing this to be skipped)
  2. Data copied from page cache to process memory (may or may not page fault)

Memory mapping is only going to beat this performance-wise because of that extra copy from the page cache to process memory. But a mere copy of a page of memory (or less) has to be done multiple times to beat the cost of setting up the mapping - probably. How many times depends on your system. Memory bandwidth, how your entire system is being used, everything. For example, if the time used by the kernel's memory management to set up the mapping wouldn't have been used by any other process anyway, the cost of creating the mapping really isn't very high. Conversely, if you have a lot of processing on your system that involves a lot of virtual memory mapping creation/destruction (i.e., lots of short-lived processes) the impact of memory mapped IO might be significant.

Then there's read() using direct IO:

  1. System call to read from disk into process memory space. (may or may not cause a page fault)

Direct IO reads are pretty much impossible to beat performance-wise. But you have to really tune your IO patterns to your hardware to maximize performance.

Note that a process can pretty much control if reading data causes a page fault for the buffer the process is using to read.

So, is memory-mapped file access faster? Maybe it is, maybe it isn't.

It depends on your access pattern(s). Along with your hardware and everything else in your IO path(s).

If you're streaming a 30 GB video file on a machine with 4 GB of RAM, and you never go back and reread any of the data, memory-mapping the file is probably the worst way to read it.

Conversely, if you have a 100 MB lookup table for some data that you randomly access billions and billions of times in your processing and enough memory that the file never gets paged out, memory mapping will crush all other access methods.

One huge advantage of memory-mapped files

Memory mapping files has a huge advantage over other forms of IO: code simplicity. It's really hard to beat the simplicity of accessing a file as if it's in memory. And most times, the difference in performance between memory-mapping a file and doing discrete IO operations isn't all that much anyway.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.