How to tell if a file is memory mapped?

Question

I am confused about memory-mapped files, so I have a couple of questions which I would be very glad if you could help me out.

1. Let's say I browse to a directory in my file system and there is a file in this directory. Is it possible that this file points to a region in the main memory, instead of pointing to a region in the disk?
2. If this is possible, is this what we call 'memory-mapped file'?
3. What would be the meaning of moving such file around the file system (that is, mving such file from a directory into another)? What I understand is, since the file is memory mapped, the process(es) interacting with the file always writes to a predefined region of the main memory, and when we open that file (for example using vim), we read that region of main memory (so, no disk is involved). Hence, no matter where we move the file, it will always work correctly right? If yes, does moving the file around the file system has any significance?
4. Is there a command which would tell if a file is memory-mapped?
5. Finally, if I open a memory-mapped file with vim, make some changes on it and save and close vim, what will happen? Will my changes simply be written to main memory? If that's the case, will other processes which use this file will see the changes I have just made? In my experience, the other processes did not see the changes I have made to the file when I made some changes on the file with vim. What is the reason for this?

Answer 1

Memory-mapped files work the other way round. Memory-mapping isn't a property of the file, but a way to access the file: a process can map a file's contents (or a subset thereof) into its address space. This makes it easier to read from and write to the file; doing so simply involves reading and writing in memory. The file itself, on disk, is just the same as any other file.

To set this up, processes use the mmap function. This can also be used for other purposes, such as sharing memory between processes.

Answer 2

A memory-mapped file is not (necessarily) backed by memory. It can perfectly live on a disk. Actually, where a file lives is not a property of the file itself but of the file system it resides in.

Mapping a file in memory is an operation a process can do to have a portion of the file loaded in memory. The result looks like a regular region of memory, except that when the process reads from or writes to this region, it actually reads from and writes to the file. If you open a file, map it to memory, write to it and save it, the modification will be done on the file, on disk (if it lives on a disk, of course).

This can be used for example when you know you have a lot of accesses to do on a file, which are not going to be sequential, be cause it can be easier and more efficient to do reads and writes in memory than to issue read, write, and llseek system calls. The only problem with this method is that you cannot really use it if the file needs to be read from or written to by several processes simultaneously. The results would be unpredictable.

I know no command which can tell you if a file is currently mapped. You can inspect the mappings of a process, though, in /proc/<pid>/maps (if your system has it).

To answer your second question, when you open a file, even if you move it in the filesystem, the processes which have opened it can still use it. What happens is that a file is not dependent from its entries in the filesystems. As long as you have a file opened, you have a "handle", a file descriptor, which lets you read from and writes to it, even if its path in the file system changes. A file disappears only when it has no entry in the filesystem and no process holds a file descriptor on it.

Answer 3

Q4: Is there a command which would tell if a file is memory-mapped?

The lsof command will show you all the files currently in use by the system. The "FD" column will contain "mem" if the file is memory mapped. So you could grep the output of this command for the filename that you are interested in.

Answer 4

You seem to be confusing memory-mapping with files in memory-residing filesystems, along with other concepts like how processes maintain access to files even as they're moved around.

I'll go question by question to see if I can clear things up.

1. Let's say I browse to a directory in my file system and there is a file in this directory. Is it possible that this file points to a region in the main memory, instead of pointing to a region in the disk?

It does point to main memory if it's on a memory-residing file-system, like procfs which is typically mounted on /proc, or sysfs which is on /sys, or tmpfs which is sometimes on /tmp.

2. If this is possible, is this what we call 'memory-mapped file'?

No. Like stephen-kitt said, "memory-mapping" refers to a way to access a file by "mapping" it on main memory and working with it there rather than reading and writing chunks at a time via functions like read() and write().

3. What would be the meaning of moving such file around the file system (that is, mving such file from a directory into another)? What I understand is, since the file is memory mapped, the process(es) interacting with the file always writes to a predefined region of the main memory, and when we open that file (for example using vim), we read that region of main memory (so, no disk is involved). Hence, no matter where we move the file, it will always work correctly right? If yes, does moving the file around the file system has any significance?

If you move it around within the same filesystem, you're really just moving around a reference, an inode from one directory to another. If there are programs that already had this file opened, they'll still be accessing the same file because they already have the inode on hand via a file-descriptor. This is what happened with the table_name.idb file you mentioned in a comment.

4. Is there a command which would tell if a file is memory-mapped?

Wossname already answered this for memory-mapped files. lsof will tell you which processes have the file memory-mapped.

To know if a file is in a memory-residing filesystem, you can use df or mount to list the filesystems and their mountpoints. You just need to know which types of filesystems reside in memory by looking them up (e.g. in wikipedia).

5. Finally, if I open a memory-mapped file with vim, make some changes on it and save and close vim, what will happen? Will my changes simply be written to main memory? If that's the case, will other processes which use this file will see the changes I have just made? In my experience, the other processes did not see the changes I have made to the file when I made some changes on the file with vim. What is the reason for this?

Personally, I haven't used the mmap function in a C program, but as I understand it from skimming man mmap and info mmap, there is no magic involved in maintaining the in-memory representation in sync. In its basic form, calling mmap copies the file contents to memory and msync is used to write it back from memory to the disk. If the on-disk file changes, there is nothing in place to detect that and automatically modify the in-memory representation in all processes that mapped it.

EDIT: Turns out that mmap() actually does try to keep the in-memory representation in sync under some conditions. If the map is only read from, it will be kept in sync even when other processes write to the file. If it's written to (by assigning to the memory region), what happens depends on which of the apparently mandatory MAP_SHARED or MAP_PRIVATE flags is provided to mmap(). If MAP_PRIVATE is provided, the map forks from the on-disk representation and stops being in sync until you use msync(). If MAP_SHARED is provided, then the updates are made visible to other processes that have the file mapped, as well as (though this isn't necesarily immediate) the on-disk representation.

I just opened vim on an existing file e, and ran the command :w, while having inotifywait -m . running in another terminal. Among some weird bits, this is the important part I got from inotifywait.

./ MOVED_FROM e
./ MOVED_TO e~
./ CREATE e
./ OPEN e
./ MODIFY e
./ CLOSE_WRITE,CLOSE e
./ ATTRIB e
./ ATTRIB e
./ DELETE e~

Vim creates a new file, and removes the old one. Why it does this instead of modifying the file is beyond the scope of this question, but the point is that this is a new file and therefore has a new inode.

Now, what do you mean by other processes using this file? If you mean processes which had the file opened while you were doing this, no they won't see the changes. This is because, although they opened a file with the same path, they aren't the same file. If you mean processes which may open the file after you did this, then yes they will see the changes. They'll be opening the new file you created.

It's important to note that though programs may seem to have a file open on the user interface, that doesn't necesarily mean that that they keep the file open in the process. Vim is an example of this, as shown above.