On Mac OS X Yosemite 10.10.5, when I try to run a calculation that needs to allocate and use 128 GB of memory (it is a command line program written in C), the kernel kills my process with extreme prejudice. This console log entry is an example of one instance:
9/25/15 7:08:40.000 PM kernel: low swap: killing pid 6202 (huffgrp)
The calculation works fine and in a reasonable amount of time when it allocates and uses 64 GB of memory. My Mac has 32 GB of RAM and beaucoup space on the hard drive. I also tried this on another Mac with 8 GB of RAM, on which the 64 GB calculation runs fine as well, taking longer of course, but the 128 GB calculation gets killed by the kernel in the same way.
By the way,
malloc() never returns an error, no matter how much space I ask for. The kernel will only kill the process once too much of that memory is actually being used by the process, resulting in lots of swapping to the hard drive.
So there appears to be a secret swap space limit somewhere between 64 GB and 128 GB.
My question is: how do I reconfigure the kernel to permit more swap space? I found a promising looking file,
/System/Library/LaunchDaemons/com.apple.dynamic_pager.plist, but I don't see the secret number in there. The man page for
dynamic_pager says that all it does is set the name and location of the swap files. There is an older version of the same man page that documents a
-S option to set the size of the swapfiles created. I tried that, requesting 160 GB swapfiles, but it had no effect. The swapfiles were still 1 GB each, and the process was still killed by the kernel.
mallocmore than you have is because the commit_limit is very high (probably infinite). Therefore the OS will allocate memory that it does not have (this is betting that the process will not use it, the os usually wins this bet). You may want to adjust the commit limit to be the memory limit, this way the process will fail early.
malloc()does that. I was deflecting possible comments on someone thinking I'm not checking the return value of
malloc(). By the way, my objective is not to fail earlier. My objective is to succeed.