Solaris swap allocation with mlocked shared memory

Question

I'm having a hard time understanding how Solaris 10 x86 swap allocations work in the presence of mlocked share memory.

I wrote a small program that:

• Gets a 1G SHM segment (shmget)
• Attaches it to the process (shmat)
• Locks a part of that (mlock, first round locks nothing, then 512M, then the whole thing)
• Write on the last, then the first, page of the SHM region
• Writes on the whole segment progressively
• Release the lock (if any)
• Detaches and cleans up.

It prints out the values of swap -s in MB after each action.

The output is as follows:

• Round 1, no mlock: (values on the right are relative to the first reading)

pid:    2221    mlock size:     0
header:      alloc  resvd   used   free [+/-  alloc  resvd   used   free]
  init:        325    130    456   2873 [+/-      0      0      0      0]
shmget:        326   1154   1480   1849 [+/-      1   1024   1024  -1024]
 shmat:        326   1154   1480   1849 [+/-      1   1024   1024  -1024]
touchE:        326   1154   1480   1849 [+/-      1   1024   1024  -1024]
touchS:        326   1154   1480   1849 [+/-      1   1024   1024  -1024]
  set1:        582    898   1480   1848 [+/-    257    768   1024  -1025]
  set2:        838    642   1480   1848 [+/-    513    512   1024  -1025]
  set3:       1093    386   1480   1847 [+/-    768    256   1024  -1026]
  set4:       1350    130   1480   1847 [+/-   1025      0   1024  -1026]
   set:       1350    130   1480   1847 [+/-   1025      0   1024  -1026]
 shmdt:       1350    130   1480   1847 [+/-   1025      0   1024  -1026]
shmctl:        326    130    456   2871 [+/-      1      0      0     -2]

Here, all good. free column was at most ~1G smaller than it was at startup.

• Round 2, mlock the first 512M of the shared segment:

 pid:    2221    mlock size:     536870912
 header:      alloc  resvd   used   free [+/-  alloc  resvd   used   free]
   init:        326    130    456   2871 [+/-      0      0      0      0]
 shmget:        326   1154   1480   1847 [+/-      0   1024   1024  -1024]
  shmat:        326   1154   1480   1847 [+/-      0   1024   1024  -1024]
  mlock:        838    642   1480   1334 [+/-    512    512   1024  -1537] <<<<
 touchE:        838    642   1480   1334 [+/-    512    512   1024  -1537]
 touchS:        838    642   1480   1334 [+/-    512    512   1024  -1537]
   set1:        838    642   1480   1334 [+/-    512    512   1024  -1537]
   set2:        838    642   1480   1334 [+/-    512    512   1024  -1537]
   set3:       1094    386   1480   1334 [+/-    768    256   1024  -1537]
   set4:       1350    130   1480   1334 [+/-   1024      0   1024  -1537]
    set:       1350    130   1480   1334 [+/-   1024      0   1024  -1537] <<<<
munlock:       1350    130   1480   1846 [+/-   1024      0   1024  -1025]
  shmdt:       1350    130   1480   1847 [+/-   1024      0   1024  -1024]
 shmctl:        326    130    456   2871 [+/-      0      0      0      0]

• Round 3, mlock the whole segment:

 pid:    2221    mlock size:     1073741824
 header:      alloc  resvd   used   free [+/-  alloc  resvd   used   free]
   init:        326    130    456   2871 [+/-      0      0      0      0]
 shmget:        326   1154   1480   1847 [+/-      0   1024   1024  -1024]
  shmat:        326   1154   1480   1847 [+/-      0   1024   1024  -1024]
  mlock:       1350    130   1480    822 [+/-   1024      0   1024  -2049] <<<<
 touchE:       1350    130   1480    822 [+/-   1024      0   1024  -2049]
 touchS:       1350    130   1480    822 [+/-   1024      0   1024  -2049]
   set1:       1350    130   1480    822 [+/-   1024      0   1024  -2049]
   set2:       1350    130   1480    822 [+/-   1024      0   1024  -2049]
   set3:       1350    130   1480    822 [+/-   1024      0   1024  -2049]
   set4:       1350    130   1480    822 [+/-   1024      0   1024  -2049]
    set:       1350    130   1480    822 [+/-   1024      0   1024  -2049] <<<<
munlock:       1350    130   1480   1846 [+/-   1024      0   1024  -1025]
  shmdt:       1350    130   1480   1847 [+/-   1024      0   1024  -1024]
 shmctl:        326    130    456   2871 [+/-      0      0      0      0]

In the last two rounds, the free column goes down an extra 512M or 1G right after the mlock, but before any page is actually touched. This "double accounting" disappears after the munlock call.

Can anyone explain this behavior? Are mlocked pages reserved both in real RAM and in backing store or something like that?

SunOS myhost 5.10 Generic_142910-17 i86pc i386 i86pc

Answer 1

If you use DISM, make sure you have ample room in your swap.

When you shmat an SHM segment with SHM_SHARE_MMU (which is not the default), you get an ISM segment, which is automatically locked in memory (not pageable). The cost of that mapping, in virtual memory, is just the size of the allocated SHM region. (Since it cannot be paged out, no need to reserve swap). mlock has no effect on these pages, they are already locked.

If you either attach the segment with SHM_PAGEABLE or with no attribute, you get a DISM segment. That one is pageable. The initial cost is the same. But, if you mlock any of that memory, the mlocked zone gets accounted again for its locked RAM usage. So the virtual memory cost is (whole mapping + mlocked zone).

It's as if, with SHM_PAGEABLE, the mapping was created "in swap", and the zones you lock require additional reservation "in ram" (the backing store for those locked pages is not released or un-reserved).

So what I was seeing is normal, as-designed.

Some information about this can be found in Dynamic SGA Tuning of Oracle Database on Oracle Solaris with DISM (280k PDF). Excerpt:

Since DISM memory is not automatically locked, swap space must be allocated for the whole segment. [...]. But it could become a problem if system administrators are unaware of the need to provide swap space for DISM.

(I was one of those unaware sysadmins...)

---

Tip: use pmap -xa to see what type of segment you have.

• ISM:

 Address  Kbytes     RSS    Anon  Locked Mode   Mapped File
...
80000000 1048576 1048576 1048576 1048576 rwxsR    [ ism shmid=0x16 ]
                                             ^      ^^^

Notice the R in the mode bits: no reservation for this mapping.

• DISM:

 Address  Kbytes     RSS    Anon  Locked Mode   Mapped File
...
80000000 1048576 1048576 1048576 1048576 rwxs-    [ dism shmid=0xa ]
                                             ^      ^^^^