1

I have huge core files, and hence set the core_pattern is set to gzip as they are written.

Later if the backtrace is to be obtained, first gunzip has to be done (and it takes a very long time!) before the core file can be fed to gdb.

I wanted to know if there is a way to pipe the core (as it is being created) to gdb (or any other program which can fetch the backtrace). I checked gdb, and there seems to be no such option (neither does readelf); before I could whip up something of this sort, I wanted to know if there is anything with the format of ELF core (on x86_64, GNU/Linux) that could prevent this from working ?

-- EDIT --

grok'd through readelf sources, and other programs which can generate backtrace, and they seem to be seek()ing through the file forwards and backwards! I am not sure if this is absolutely necessary, or if its possible to read and gather all the needed info in a single pass (since I want to read from a pipe, seek is not an option!)

1

A kernel coredump handler can generate the backtrace from the zombie process, without writing the coredump to disk first.

I assumed that ABRT could do this. However, it seems that there is a patch, but it has not been merged yet:

  • Thank you! now I know, I'm not the only one who's faced this issue! :) – vyom Jul 19 '17 at 3:25
0
+50

Generating a backtrace requires fetching memory contents by address. In a core file each virtual address whose contents are included in the core is at a particular offset in the core file. When code which generates a backtrace performs a seek, the target of that seek is the offset for the memory contents for an address which maps to that offset.

The obstacle to treating a core file as a stream is that there is no reason to expect the offsets corresponding to addresses whose contents are being fetched to increase with each fetch. A backwards seek simply means that the current fetch is for contents whose offset is smaller than the offset of the previous fetch.

Perhaps the simplest scenario for constructing a backtrace from a core file is a core file generated by code which consistently uses a frame pointer. In this scenario the memory fetches are from the stack, and the frame pointer values, one per stack frame, provide what amounts to a linked list of stack frames, in which the next word of each stack frame is a return address (which, in the presence of appropriate symbols, can be mapped to a function name and offset).

Increasing addresses in a typical core file have increasing offsets within the file, and a typical backtrace represented with a linked list of frame pointer values has a larger frame pointer address for each successive frame. A simple backtrace from a core file from code which uses a frame pointer could begin by locating the stack (from the value of the stack pointer register), and traversing the linked list of frame pointer values, displaying the return address associated with each frame. This approach would involve a fair amount of buffering, mainly of the program segment headers near the beginning of the core file, which are used to map virtual addresses to file offsets. The worst of the inefficiency in this approach would presumably be ignoring all of the uncompressed data between the segment headers (and the note used to obtain register values) and the stack of the relevant thread.

Generating a backtrace from code which does not use a frame pointer is more complicated, but that might also be possible if enough data is buffered. It's more likely to be feasible if the correct symbols are available, more so if debug symbols are available. As with the frame pointer scenario, fetching data from the relevant stack is likely to be essential.

Rather than beginning with the hope of treating a core file as a stream, perhaps the best approach here is to change the way space savings are achieved for huge core files. The fundamental problem with gzip output is that it does not provide random access. A better choice for a space efficient representation of a core file is a format like squashfs, in which random access is provided by compressing constant size chunks, and those chunks are indexed, so that obtaining data for a specified offset consists first of locating and uncompressing the appropriate chunk, and then seeking within that uncompressed data.

The immediate challenge with using squashfs in core_pattern is that there is no obvious way to invoke mksquashfs on standard input. Perhaps the simplest way to work around this is to allow a core file to exist in its uncompressed form long enough to invoke mksquashfs. A more space efficient approach might involve writing code which can generate squashfs format from standard input; this is likely to be straightforward if mksquashfs never has a reason to seek backwards in a file it includes in a squashfs image (I don't know this for sure, but my understanding of squashfs format suggests that it is at least a possibility).

Assuming you have a squashfs image containing a huge core file, one way to feed it to a tool which generates a backtrace is to simply mount the image and provide the relevant pathname within that mount point, but it is also feasible to modify applications to handle squashfs format natively. One advantage of this approach is that any other operation on a core file beyond a simple backtrace is also available (a likely next step in a core analysis is generating multiple backtraces, one per thread).

  • Thanks for the detailed reply, squashfs etc might not be an option for me, but will sure explore it. For the backtrace part, from your response what I can understand is - it might be be possible, but not always, is that correct? – vyom Jul 4 '17 at 13:46
  • It might indeed be possible, particularly considering what I believe is an error in my answer. I'll edit my answer. – Eirik Fuller Jul 4 '17 at 13:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.