0

I've been working with my client for some time now to try to determine some strange behaviours with core file generation on their Solaris SPARC systems. Their current version of Solaris is 11.4-11.4.44.0.1.113.4. Applications are built with GCC 11.2 with -g. Applications and core files are debugged with GDB 10.2 and pstack.

I provided the client with a special tool to help with the analysis and diagnosis of this problem. All the application does is to generate a core file due to a segmentation fault.

The problem: Core files generated on the client's systems won't show stack, variable, or memory information using GDB. But core files generated using the exact same executable on any of my development systems will generate core files that can be debugged.

At first I thought it may have to do with Solaris 11.4 patch incompatibility. I went through trying every Solaris patch level from 11.4.39 though 11.4.47 against the core file sets I have from the client and it didn't make any significant difference.

For example from development systems a core file will generate this in GDB.

. . .

Reading symbols from core_dump_tool.exe...

[New LWP 1]

[Thread debugging using libthread_db enabled]

[New Thread 1 (LWP 1)]

Core was generated by `./core_dump_tool.exe'.

Program terminated with signal SIGSEGV, Segmentation fault.

#0 0xfe3cdc80 in _malloc_unlocked () from /usr/lib/libc.so.1

[Current thread is 1 (LWP 1 )]

(gdb) where

#0 0xfe3cdc80 in _malloc_unlocked () from /usr/lib/libc.so.1

#1 0xfe3cd9d4 in do_malloc () from /usr/lib/libc.so.1

#2 0x00011a40 in BufferOverflow (iDataLength=100, cpData=0xffbff140 "0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789") at core_dump_tool.c:82

#3 0x00011f80 in main () at core_dump_tool.c:232

Backtrace stopped: previous frame inner to this frame (corrupt stack?)

(gdb) quit

. . .

When the exact same application is run on the client's systems, the core file generated will produce this in GDB

. . .

Reading symbols from core_dump_tool.exe...

[New LWP 1]

[Thread debugging using libthread_db enabled]

[New Thread 1 (LWP 1)]

Core was generated by `core_dump_tool.exe'.

Program terminated with signal SIGSEGV, Segmentation fault.

#0 0xfeb6daf0 in _malloc_unlocked () from /usr/lib/libc.so.1

(gdb) bt

#0 0xfeb6daf0 in _malloc_unlocked () from /usr/lib/libc.so.1

#1 0x00022514 in ?? ()

Backtrace stopped: previous frame identical to this frame (corrupt stack?)

(gdb) quit

. . .

With regards to the client's systems and this issue with the core files generated on their systems.

  • The issue first presented itself in December of 2021. I was able to narrow the core file issue starting somewhere between September of 2021 and early December 2021.
  • The client doesn't know what changes where made to the production machines during this time. They can't find any records and the admins have since left.
  • I've requested the coreadm and dumpadm configurations from the client for the impacted systems, but haven't got any information back.

If anyone has any thoughts or ideas on what the client may have done to their systems that are impacting the core files generated, it would be really appreciated.

1 Answer 1

1

It seems likely that your client has limited core file size on the system(s) in question.

Solaris writes the stack information to a core file last, so if the core file size is limited by any resource limit, the stack information won't be written to the core file. And the core file will be effectively useless for debugging.

There are only two useful core file size limit settings on Solaris: 0 to prevent core file creation, and unlimited so the core files that are generated are actually useful.

And a note about coreadm settings - if you change the core file pattern from the default of core in the process's current working directory in order to do things like capture process name, pid, time and similar data in the core file name and/or to keep more than the last core file named core around, or use global core file settings to capture all core files to common location, now you have to actively manage core files LEST THEY FILL UP YOUR STORAGE.

What happens if a process on a critical system gets into a state where it rapidly fails over and over and over and over, generating a new core file each time? Disks fill up, system goes down, all because you set a core file name pattern using coreadm to keep all core files.

Oooops.

Never tell a client to set a core file name pattern to keep all core files around (or configure your own systems to do so) without telling them if they do that they'll have to actively manage core files on those systems.

And "actively manage core files" does not mean "run a cron job once a day" - whatever you put in place has to be able to handle that runaway process generating 500 core files per minute.

Or at least tell them to use global core files written to a dedicated file system that can't fill up the rest of the system's storage so that if that dedicated core-file-collection file system does fill up it does no harm to the rest of the system.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .