16

On Mac OS X Yosemite 10.10.5, when I try to run a calculation that needs to allocate and use 128 GB of memory (it is a command line program written in C), the kernel kills my process with extreme prejudice. This console log entry is an example of one instance:

9/25/15 7:08:40.000 PM kernel[0]: low swap: killing pid 6202 (huffgrp)

The calculation works fine and in a reasonable amount of time when it allocates and uses 64 GB of memory. My Mac has 32 GB of RAM and beaucoup space on the hard drive. I also tried this on another Mac with 8 GB of RAM, on which the 64 GB calculation runs fine as well, taking longer of course, but the 128 GB calculation gets killed by the kernel in the same way.

By the way, malloc() never returns an error, no matter how much space I ask for. The kernel will only kill the process once too much of that memory is actually being used by the process, resulting in lots of swapping to the hard drive.

So there appears to be a secret swap space limit somewhere between 64 GB and 128 GB.

My question is: how do I reconfigure the kernel to permit more swap space? I found a promising looking file, /System/Library/LaunchDaemons/com.apple.dynamic_pager.plist, but I don't see the secret number in there. The man page for dynamic_pager says that all it does is set the name and location of the swap files. There is an older version of the same man page that documents a -S option to set the size of the swapfiles created. I tried that, requesting 160 GB swapfiles, but it had no effect. The swapfiles were still 1 GB each, and the process was still killed by the kernel.

Improve this question
7
  • I also posted this question on apple.stackexchange.com, but there has not been any response there.
    – Mark Adler
    Sep 27, 2015 at 15:41
  • 1
    The reason that you can malloc more than you have is because the commit_limit is very high (probably infinite). Therefore the OS will allocate memory that it does not have (this is betting that the process will not use it, the os usually wins this bet). You may want to adjust the commit limit to be the memory limit, this way the process will fail early.
    – ctrl-alt-delor
    Sep 27, 2015 at 20:38
  • I can not see why you thing that paragraph 5 is true, it does not follow from paragraph 4 (though you start it with “so”). Also have you tried adding lots of swap? (the question is not clear on this, though I suspect the answer is yes).
    – ctrl-alt-delor
    Sep 27, 2015 at 20:41
  • @richard Yes, I know why malloc() does that. I was deflecting possible comments on someone thinking I'm not checking the return value of malloc(). By the way, my objective is not to fail earlier. My objective is to succeed.
    – Mark Adler
    Sep 27, 2015 at 23:38
  • @richard Because the kernel message says "low swap". As to "adding lots of swap", no I haven't done that because I don't know how. That is precisely this question. How do I add swap space? Of course, the kernel adds swap space automatically, but only up to a limit. Hence the root of question: How do I increase the kernel's swap space limit?
    – Mark Adler
    Sep 27, 2015 at 23:41

2 Answers 2

Reset to default
5

Not the answer you asked for, but if you create your own file of an appropriate size, mmap it into your process, and then run your calculation in this address space it should have the same effect as a swap file and you're guaranteed to have the space, as opposed to competing with other processes for available RAM/swap.

It might also be slower, depending on how often you overwrite the data, but should be much more portable.

Improve this answer
2
  • 3
    Great idea! I had high hopes. Alas, they were dashed against the harsh shore of reality. The process was still killed by the kernel for large amounts of swapping, just as before ("low swap" console message), even though it was using a file I had created to use as virtual memory by mmap().
    – Mark Adler
    Oct 1, 2015 at 5:48
  • Sorry, most of my experience is with Linux. But, maybe the problem is that the kernel isn't flushing back to the file and holding it in RAM instead. What if you periodically call msync() to force it to do so?
    – dataless
    Oct 1, 2015 at 18:25
1

My mac info is quite dated there may no longer be the kernel magic to do this. As such I suggest using Linux for this program, where you can mount either a folder or partition as swap very simply.

Eliminate the uncertainties of using a dynamic swap folder. Create a physical swap partition (just an empty unformatted partition, with disk table type code for swap, in Linux it is hex code 0x82.) Then:

  1. Edit the /etc/rc file find the swap section comment out all of it. add the line mount -vat swap
    • means mount all swap partitions in /etc/fstab
  2. run pdisk /dev/disk? -dump
    • "?" is the disk number of your hd record the number next to the swap partition
  3. Edit or create /etc/fstab (it probably doesn't exist) add a line like /dev/disk?s?? none swap sw 0 0
    • ? is your disk.
    • ?? is the number of the swap partition.
  4. reboot.
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.