Questions tagged [mmap]
All about using memory mapped files. Questions on programming should be asked on Stack Overflow SE.
80
questions
6
votes
1
answer
578
views
Does mmap() update the page table after every page fault?
Based on my research on mmap(), I understand that mmap uses demand paging to copy in data to the kernel page cache only when the virtual memory address is touched, through page fault.
If we are ...
- 126k
25
votes
6
answers
12k
views
what is the purpose of memory overcommitment on Linux?
I know about memory overcommitment and I profoundly dislike it and usually disable it. I am not thinking of setuid-based system processes (like those running sudo or postfix) but of an ordinary Linux ...
- 99
1
vote
0
answers
26
views
Which consistency guarantees do POSIX shared memory objects have?
On Linux, POSIX shared memory objects [1] use a tmpfs via /dev/shm. A tmpfs in turn is said to "live completely in the page cache" [2] (I'm assuming swap has not been enabled). I am ...
3
votes
1
answer
80
views
How is 4kB RSS possible in Linux 4.x?
I have been the dev/maintainer of an open source IRC bot since the late 90s. The goal was always to make it as versatile & useful as possible in a small memory footprint.
During the 2000s I also ...
- 7,981
1
vote
0
answers
111
views
Process memory layout - difference between heap, data and mmap areas
I see in the web many conflicting or unclear descriptions of the memory layout of a Linux process. Usually the common diagram looks like:
And a common description would say that:
The data segment ...
- 72.9k
0
votes
0
answers
53
views
How to read watchdog registers on x86 Linux?
I want to read the Intel iTCO watchdog registers on my Intel Lynx Point system. I found the watchdog here:
[ 5598.341020] iTCO_wdt iTCO_wdt.1.auto: Found a Lynx Point TCO device (Version=2, TCOBASE=...
- 153
0
votes
1
answer
46
views
Can mmap be used to create a file which references memory subset of another file?
I'm interested in writing a program that can create two files, second file would be a "view" of first file and if modified, the first file would also be modified. Is this possible to do with ...
- 437k
0
votes
0
answers
40
views
Searching a 1T block device for a specific byte sequence at a specified offset
I'm performing data recovery after an accident with dd. In the longer term, I'll need to use some recovery tools to try and repair the file system
In the meantime, there's an image on the system that ...
0
votes
1
answer
82
views
How to measure mmap I/O latency?
I have an application which appears to be slowing/blocking at the same time there's a lot of disk I/O going on, so I suspect it's I/O operations within the application which are blocking. I can't ...
- 29.4k
1
vote
0
answers
308
views
Linux HugeTLB: What is the advantage of the filesystem approach?
Moved Post Notice
I just moved this question (with slight modifications) from a StackOverflow question (which I have deleted, since cross-posting is strongly discouraged), which has not been answered ...
- 191
0
votes
1
answer
102
views
Mapping Segment of Guest RAM to host file, in PPC QEMU
My desire is conceptually simple, I have a file (really a PCIe resource file from /sys/bus/pci/device/.... but that isn't too relevant) on the host that I want to make available somewhere in guest ...
- 381
2
votes
1
answer
2k
views
Mapping guest RAM to file in qemu
We're emulating a Cortex M3 cpu and would like to pass some parameters to the guest during run-time. The simplest idea seems to be to write directly to some memory area. I tried simply adding -mem-...
- 381
1
vote
1
answer
123
views
Partial fsyncs when writing to block device
I'm writing my own data store directly on top of a block device. To ensure durability I want to sync to disk. But here's the thing: I want to sync only part of it.
I'm keeping a journal for crash ...
- 50.3k
1
vote
1
answer
2k
views
Is it possible for two processes to use the same shared-memory without resorting to a file to obtain it, be it a memory-mapped file or /dev/shm file?
I'm curious because today the only way I know how to give two different processes the same shared-memory is through a memory-mapped file, in other words, both processes open the same memory-mapped ...
- 29.4k
0
votes
1
answer
36
views
What Linux distros support mmap_shared?
I found a man page for a function called mmap_shared on die.net: https://linux.die.net/man/3/mmap_shared
It looks like a convenience wrapper function around mmap, however it doesn't seem to be ...
- 29.4k
1
vote
0
answers
80
views
Impossible to use memory allocated to buffers with transparent huge pages (process killed)
I have 1TB of RAM, 900GB of which I need to allocate and use in a process (I have complete control on the hardware and I'm working on bare metal). I allocate 900GB of memory using mmap() (private, ...
- 251
0
votes
1
answer
335
views
How do I open a mmap of size of a TB
I need to open a huge memory map. The file is one terabyte.
I however am getting an errno: ENOMEM 12 Cannot allocate memory. I don't get what is holding me up. Requesting the RLIMIT_AS results in the ...
- 1,001
1
vote
0
answers
59
views
Can kernel's flushing/syncing of mmaped file interrupt a write() call?
I know there is (shamefully) no way to prevent kernel from flushing dirty pages of files mmap-ed with MAP_SHARED, so to take the control back, I wanted to map a file with MAP_PRIVATE, then when I want ...
- 214
27
votes
2
answers
29k
views
How does memory mapping a file have significant performance increases over the standard I/O system calls?
Operating System Concepts says
Consider a sequential read of a file on disk using the standard
system calls open(), read(), and write(). Each file access requires
a system call and disk access.
...
- 2,655
4
votes
1
answer
5k
views
Does mmap allow creating a mapping that is much larger than the amount of physical memory?
Obviously, mmap is limited by the size of the largest available block of virtual address space. But let's assume we have a 64-bit system, where this is almost unlimited for most purposes. Let's also ...
- 2,655
0
votes
1
answer
166
views
Why Disk stats show many read operations when I measure NVME squance write with fio and mmap as ioengine
Here is my fio configure and report:
# cat fio-write.fio
[global]
name=fio-seq-writes
filename=test
rw=write
bs=1M
direct=0
numjobs=1
[file1]
size=1G
ioengine=mmap
iodepth=1
# fio --version
fio-3.30
...
1
vote
1
answer
139
views
Is there a way to open a file and prevent flushing updates to hardware unless I command it to?
I know for kernel functions open(), mmap() and msync() and what they do. But no matter what combination of flags I attempt to use, the kernel will either still flush updates using some of its own time ...
- 214
2
votes
0
answers
138
views
Question about transfer data between specific physical memory and NVME
I'm trying to transfer data from specific reserved physical memory to my NVME, IIUC, there are 3 steps:
In the kernel driver, the reserved-memory (specified physical address and size in dts) is ...
- 387
1
vote
0
answers
250
views
EXT4 journal=data and mmap
Does enabling data=journal for ext4 make any difference when using mmap() to update a file?
The ext4 man page says:
journal
All data is committed into the journal prior to being written into the main ...
- 150
16
votes
1
answer
22k
views
Understanding mmap
I was going through documentation regarding mmap here and tried to implement it using this video.
I have a few questions regarding its implementation.
Does mmap provide a mapping of a file and return ...
- 8,259
0
votes
1
answer
438
views
In what segment does mmap allocate memory
I thought first that it was the heap, but it seems to allocate memory in a different place. radare2 tags it as folowing:
0x00007fb07dacd000 - 0x00007fb07dace000 - usr 4K s rw- unk2 unk2
Since it ...
- 437k
2
votes
2
answers
823
views
Write in /dev/mem without using mmap
It is possible to write on /dev/mem without using mmap?
I'm enabling pull-up resistors on a Raspberry Pi inside an LKM and the function void *mmap (caddr_t addr, size_t len, int prot, int flags, int ...
- 2,959
0
votes
1
answer
67
views
Is the memory usage of typical software exaggerated?
Suppose multiple processes are using the same shared library (such as Gtk). Presumably, they mmap the library file, and physically, the RAM is shared?
However, the size of the library gets added to ...
- 6,785
1
vote
0
answers
191
views
Howto enable Nginx memory caching with mmap()
I am a newbie nginx admin. I used nginx-quic to get features like HTTP/3 over QUIC. However, HTTP/3 perfoms 2x to 3x slower than HTTP/2, so I am trying to optimize the experimental server. (The test ...
- 153
1
vote
1
answer
2k
views
Do docker containers share RAM for files memory mapped from the same layer but a different image?
I'm not 100% certain about whether this is a U&L question or a SO question. On balance I'm posting it on U&L as it's OS related.
Background
As far as I know, Linux will load shared libraries (...
- 437k
1
vote
1
answer
1k
views
mmap(): Is it possible to prevent writing back to file with MAP_SHARED flag?
As I understand, 'MAP_SHARED' flag in mmap() shares any changes made by a process to the memory map immediately with other processes and eventually writes the changes back to the file. Is it possible ...
- 13
1
vote
0
answers
161
views
linux enable large page management
I am doing some experiments. Some huge pages (2MB) are used in the experiment, so that the 21-bit page offset can remain unchanged when performing virtual address translation.
I found some methods on ...
- 193
6
votes
1
answer
1k
views
Is it possible to disable unnecessary disk writes to mmap files on linux?
I would like to know if there is a way to prevent Linux from periodically syncing mmap'd files to disk, while still allowing the OS to write back when physical memory gets tight.
I am writing ...
- 141
1
vote
0
answers
98
views
Is there a way to tell how much memory is file-backed on Linux process?
I'm looking to see how much of my memory is file-backed on a process. Unfortunately proc/pid/smap_rollup shows stats for how much memory is being used, but it doesn't seem to differentiate between ...
- 111
0
votes
1
answer
1k
views
Do anonymous memory mapping and shared memory allocate space from physical memory only?
In Linux,
when using mmap() for anonymous memory mapping, or using malloc(), do they allocate "space" from only physical memory, or either physical memory or swap or their combination? (I ...
- 437k
-1
votes
1
answer
825
views
What is the path of data transfer when using memory mapped file?
Is there any difference between the data transfer paths using read()/write() and using mmap() on a file?
What does "kernel" mean in https://stackoverflow.com/a/41419353?
mmap doesn't ...
- 437k
3
votes
1
answer
1k
views
mmap a file vs mmap in malloc
I'm kind of confused by mmap.
Well, I know that when we malloc a big size of memory, we will invoke the function mmap, which will allocate an area in memory. In this case, mmap just allocate some ...
- 437k
5
votes
2
answers
4k
views
What is the behaviour of a file-backed memory map when reading from or writing to an address larger than the length of the file?
I'm trying to figure out whether it would be undefined behaviour to open a memory map to a file with a very large length. My use-case is that I want to be able to see new data when the file is ...
- 617
0
votes
0
answers
215
views
SIGBUS immediately after successful mremap for shared memory
I am using POSIX shared memory. Initially I map four shared memory buffers. One of them exceeds its allocated size so I call mremap:
void * remap_shm(void *old_address, size_t old_size, size_t ...
- 125
2
votes
1
answer
1k
views
How to revoke write permissions on a shared memory object s.t. subsequent writes to aleady mapped pages by other processes will fail?
Is there any way to revoke write permissions to shared memory by the process who created that shared object, s.t. any other process who has mapped the shared memory to its virtual space with write ...
- 18.7k
1
vote
1
answer
419
views
Memory map to process for a large code segment
I know that pmap of a process shows how the memory mapping is done. For example, the first lines of pmap output shows the memory mapping to the text segment of the process' executable. Assume I have a ...
- 437k
1
vote
0
answers
271
views
php 7.3 mmap munmap too much calls\time on Ubuntu 18+
Good day everyone. Here i have problem with after migrating highload backend php server to new one. ( Dedicated ) . For this point i started comparing ubuntu 16 and 20 ( 18 the same ) .
I bought 2 ...
- 11
5
votes
2
answers
15k
views
How to use dd if=/dev/mem in place of devmem ?
It seems like
root@testbox:~# dd if=/dev/mem bs=1 count=4 skip=2149646336 | hd
should be nearly equivalent to
root@testbox:~# devmem 2149646336 32
But, while devmem works great, that dd invocation ...
- 111
2
votes
2
answers
1k
views
What is lazy swap reservation?
I am reading the Linux Progamming Interface.
49.9 MAP_NORESERVE and Swap Space Overcommitting
Some applications create large (usually private anonymous) mappings,
but use only a small part ...
CommunityBot
- 1
1
vote
0
answers
3k
views
Mmap failed with operation not permitted
I've compiled application on Linux, but I got this error
wiringPiSetup: mmap (GPIO) failed: Operation not permitted
After that:
I've added iomem=relaxed to grub
cat /proc/cmdline
BOOT_IMAGE=/...
CommunityBot
- 1
1
vote
1
answer
1k
views
Weird major page fault number when reading sequentially / randomly in mmap region
I'm following this answer, trying to generate some major page faults with mmap:
#include <fcntl.h>
#include <stdio.h>
#include <sys/mman.h>
#include <sys/stat.h>
int main(...
- 201
2
votes
1
answer
777
views
User-space swap on mmap()-ped files
Imagine a system with very small ram (nearly embedded). It is enough - for nearly all tasks, but with an exception. There is a task X, which requires a huge amount of RAM what the machine has not.
...
- 201
0
votes
2
answers
1k
views
Does a forked process copy all mapped memory when writing?
I understand that a forked process maps the parent's memory and copies it when writing to it. Does it copy just what it needs written or does it copy the entire mapped memory ?
- 201
2
votes
1
answer
2k
views
Mmaping tremendously large files
I have a very large disk drive (2TB), but not very much RAM (8GB). I'd like to be able to run some big data experiments on a large file (~200GB) that exists on my disk's filesystem. I understand that ...
1
vote
1
answer
793
views
Pagemap on memory mapped devices not working
I am trying to find the physical addresses of heap variables, stack variables and memory mapped peripheral addresses using the /proc/{pid}/pagemap file using the steps detailed in the file: http://lxr....
- 201