0

I've observed that I can create a shared memory object and give it ostensibly any size with ftruncate, regardless of the resource limits on my system. The code below sets the size to 262 TB, and indeed stat() reports that is the size. However, 262 TB is greater than my /dev/shm mount, and I definitely do not have 262 TB of memory or swap available on the system anyway. I expected an OOM error, but I did not see my system memory usage change significantly.

What's going on here? Why does ftruncate succeed with sizes beyond my system's resource limits? I assume there is some special interaction here between ftruncate and shared memory objects (or perhaps more generally, with tmpfs).

import os
import _posixshmem
try:
    fd = _posixshmem.shm_open("test", os.O_CREAT | os.O_EXCL | os.O_RDWR, mode=0o600)
    os.ftruncate(fd, 52428800 * 5000000)
    print(os.stat(fd))
finally:
    _posixshmem.shm_unlink("test")
os.stat_result(st_mode=33152, st_ino=244, st_dev=24, st_nlink=1, st_uid=1000, st_gid=100, st_size=262144000000000, st_atime=1693167355, st_mtime=1693167355, st_ctime=1693167355)

2 Answers 2

1

The ftruncate system call changes the length of a file, not its size. If you use ls -s on the file before and after ftruncate, you will see the actual size has not changed.

What you have done is to create a file with a hole in it. The hole is unallocated space that has no blocks (or in this case, memory pages) assigned to it. With either a file or a shared memory segment in this state, you can read data from the hole (and get all zeros) but a page won't actually be allocated until you write to it.

And, of course, if you tried to write to all of those pages, the system would likely run out of memory and crash.

2
  • Thanks! I was tying this issue too closely to shared memory, but searching more generally about ftruncate and memory allocation led me to discovering the concept of a "sparse file", which is what's going on here. There is a similar question here that I unfortunately missed during initial research.
    – bgfvdu3w
    Aug 28, 2023 at 19:01
  • A sparse file is a file with one or more holes in it.
    – user10489
    Aug 28, 2023 at 22:59
0

Any physical memory is not needed until you write something into that memory. Therefore your action does not have to appear in stats.

More precisely, there might be some memory needed to mark all pages as not-yet used, but 1. this possibly does not show in stats 2. this may be avoided if instead, a note is made that number of last page (of that file) you ever used is ... well, zero.

It is not very portable to rely on such a behaviour.

3
  • Right, it makes sense that memory isn't used if nothing is written. But what is ftruncate actually doing to the shared memory object then, and why does stat() report a huge size?
    – bgfvdu3w
    Aug 28, 2023 at 0:22
  • The huge items in memory stats will be virtual memory or virtual address space. And in your code, you stat a file descriptor. If that is backed by something similar to a file, it would be supposed to have well defined length and well defined content. What you just have set is the length, it it is 52428800 * 5000000 = 262144000000000.
    – minorChaos
    Aug 28, 2023 at 0:43
  • Note that stat returns size and length separately. The first is number of allocated blocks, the second is offset of the last byte in the file. (These are basically the same for files without holes.)
    – user10489
    Aug 28, 2023 at 23:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .