Trying to figure out why running some software that I am developing causes my Mac to do a panic restart every 16 h or so always with an "os_refcnt" kernel variable overflow when running tcsh scripts. In test cases, restarting & rerunning the code from where it leaves off gives correct results. At first I thought it was references to shell variables, but unsetting all such variables in several different ways, after the scripts were finished using them, had no effect.
The big question is what type of events are the "references" that "os_refcnt" is counting?
Why could the count value be overflowing? Why is it not resetting? Could the overflow in the number of "references" be due to a "race condition" preventing the memory from resetting due to there being many simultaneous threads? (Program runs simultaneously on different data on 12 independent branches, the max that could be used while still causing the runtime to be shorter.)
Mac Tech support did not offer any good suggestions.
Other than UNIX-like system calls, the scripts use programs compiled with compilers from the GNU project, in case that has something to do with this. The main program, in terms of CPU time, is written and compiled by the scripts. This program holds a big constant data array that is calculated by another program in the script. The 4 input values are 4 integers that specify the indices of 2 elements of the array and the program multiplies together the values of the 2 elements.
Since the error in the "panic report window" is with an overflow of "os_refcnt", an integer-valued system variable used by kernel, and since I had no idea about what kernel was doing, I did a Google search & found that one of the things that kernel does is to slow jobs down to control the CPU temperature. Seemed odd, so to check the temperature, I downloaded a CPU temperature display app and found that the CPU temperature was fairly hot, hovering between 84°C and 86°C just before the crash. Got a few extra external fans to blow at the Cylinder-Shaped MacPro "Stack" and was able to keep the maximum sustained T at 82-84°C most of the time. At 16 h the 2 jobs had synched so that there was little CPU down time between intrajob segments and the CPU's may have had reason to be slightly hotter. However, using the external fans, even at the slightly lower temperature, the computer crashed after 16.5 h, which does not seem to be a significant difference from 16 h. If cooling is the issue, I might be able to add intentional delays to try to cool down the CPUs to see if that helps.
I tried Googling about os_refcount, but no luck. I have made a lot of other guesses about what the reference count overflow might be that is causing the panic restart. I set the scripts so that all secondary & tertiary scripts called within the main script are implemented using the "tcsh" command instead of the "source" command so that variable values are supposed to be "forgotten" on exiting the secondary scripts and I also explicitly added "unset" on each of the shell variables I am using.
Does anyone else have an idea about what may be causing the kernel variable "os_refcnt" to overflow consistently? (Was thinking of translating the scripts that I am using from tcsh to bash to see if this would improve things, but that also seems a little far fetched, and wanted to check if it was a reasonable thing to try before undertaking all of the required debugging.) For now, I just restart the computer every 16 h to get the results that I need.
The problem has been here for a while. Right now running on MacOS10.15.4 Catalina. The same program would not cause the OS to crash as often on older Mac's, but instead would cause 1 in ~10^8 "egrep -c" commands to return NaN (bad values). These NaN values could be caught within the running scripts, and then when the egreps were re-run immediately afterward, they would give the right numbers. The older Mac's were about 5 - 20 times slower though. They also had the more typical rectangular-box shaped stacks instead of the cylinder, and also had audibly whirry fans. The older Macs did crash after about 1 week or 2, which might be after about the same number of program cycles, so maybe I am seeing a continuation of the same problem. (To try to avoid the crashing, I tried other versions that used "awk" equivalents instead of the "egrep"; THE SAME CRASHING happened with either "awk" or "egrep", but interestingly on some of the OS versions, "awk" version was a lot faster, on other versions "egrep" was a lot faster by factors of 2x to 3x in runtime difference--also other kinds of Mac hardware used besides the using the older OS's.)
When I called Apple Tech Support for help, the answer was "It is crashing because of your program, i.e. 3rd Party Software" but little about what I could do to prevent this, other than "Don't run your program."