Generating a backtrace requires fetching memory contents by address. In a core file each virtual address whose contents are included in the core is at a particular offset in the core file. When code which generates a backtrace performs a seek, the target of that seek is the offset for the memory contents for an address which maps to that offset.
The obstacle to treating a core file as a stream is that there is no reason to expect the offsets corresponding to addresses whose contents are being fetched to increase with each fetch. A backwards seek simply means that the current fetch is for contents whose offset is smaller than the offset of the previous fetch.
Perhaps the simplest scenario for constructing a backtrace from a core file is a core file generated by code which consistently uses a frame pointer. In this scenario the memory fetches are from the stack, and the frame pointer values, one per stack frame, provide what amounts to a linked list of stack frames, in which the next word of each stack frame is a return address (which, in the presence of appropriate symbols, can be mapped to a function name and offset).
Increasing addresses in a typical core file have increasing offsets within the file, and a typical backtrace represented with a linked list of frame pointer values has a larger frame pointer address for each successive frame. A simple backtrace from a core file from code which uses a frame pointer could begin by locating the stack (from the value of the stack pointer register), and traversing the linked list of frame pointer values, displaying the return address associated with each frame. This approach would involve a fair amount of buffering, mainly of the program segment headers near the beginning of the core file, which are used to map virtual addresses to file offsets. The worst of the inefficiency in this approach would presumably be ignoring all of the uncompressed data between the segment headers (and the note used to obtain register values) and the stack of the relevant thread.
Generating a backtrace from code which does not use a frame pointer is more complicated, but that might also be possible if enough data is buffered. It's more likely to be feasible if the correct symbols are available, more so if debug symbols are available. As with the frame pointer scenario, fetching data from the relevant stack is likely to be essential.
Rather than beginning with the hope of treating a core file as a stream, perhaps the best approach here is to change the way space savings are achieved for huge core files. The fundamental problem with
gzip output is that it does not provide random access. A better choice for a space efficient representation of a core file is a format like
squashfs, in which random access is provided by compressing constant size chunks, and those chunks are indexed, so that obtaining data for a specified offset consists first of locating and uncompressing the appropriate chunk, and then seeking within that uncompressed data.
The immediate challenge with using
core_pattern is that there is no obvious way to invoke
mksquashfs on standard input. Perhaps the simplest way to work around this is to allow a core file to exist in its uncompressed form long enough to invoke
mksquashfs. A more space efficient approach might involve writing code which can generate
squashfs format from standard input; this is likely to be straightforward if
mksquashfs never has a reason to seek backwards in a file it includes in a
squashfs image (I don't know this for sure, but my understanding of
squashfs format suggests that it is at least a possibility).
Assuming you have a
squashfs image containing a huge core file, one way to feed it to a tool which generates a backtrace is to simply
mount the image and provide the relevant pathname within that mount point, but it is also feasible to modify applications to handle
squashfs format natively. One advantage of this approach is that any other operation on a core file beyond a simple backtrace is also available (a likely next step in a core analysis is generating multiple backtraces, one per thread).