How to view core files for debugging purposes in Linux?

Question

I want to view the contents of a core file while debugging a program. How can I view the contents of a core file?

Answer 1

GDB minimal runnable example

GDB had been previously mentioned at: https://unix.stackexchange.com/a/89934/32558 Consider upvoting that answer.

simple.c

int myfunc(int i) {
    *(int*)(0) = i;
    return i - 1;
}

int main(int argc, char **argv) {
    (void)argv;
    int i = argc * 2;
    int ret = myfunc(i);
    return ret;
}

Compile, and run to generate core:

gcc -ggdb3 -std=c99 -Wall -Wextra -pedantic -o simple.out simple.c

To generate the core file, we first have to run in the current terminal:

ulimit -c unlimited

which means "dump core files without any size limit". This exists because core files contain the entire memory of the crashing process, and so they could be very large.

Tested as of Ubuntu 16.04, you have to remove a pre-existing core file (TODO mandatory? I forgot):

rm -f core

Tested as of Ubuntu 22.04, you need to fight against apport to get your core file: https://askubuntu.com/questions/1349047/where-do-i-find-core-dump-files-and-how-do-i-view-and-analyze-the-backtrace-st/1442665#1442665 e.g. with:

echo 'core' | sudo tee /proc/sys/kernel/core_pattern

Then we run the program:

./simple.out

and the terminal contains:

Segmentation fault (core dumped)

The core file has been generated. On Ubuntu 16.04 the file is named just:

core

On Ubuntu 22.04 after echo 'core' | sudo tee /proc/sys/kernel/core_pattern the file is named as:

core.<pid>

where PID is the process ID, a number, e.g.:

core.162152

I think this is because of a Linux kernel update that started adding the .pid suffix. TODO confirm.

We can now use the core file as either

gdb simple.out core
gdb simple.out core.162152

and now we enter a GDB session which is exactly as things would have been when the program crashed, except of course we can't "continue running" as the program is about to end:

#0  0x0000557097e0813c in myfunc (i=2) at simple.c:2
2           *(int*)(0) = i; /* line 7 */
(gdb) bt
#0  0x0000557097e0813c in myfunc (i=2) at simple.c:2
#1  0x0000557097e0816b in main (argc=1, argv=0x7ffcffc4ba18) at simple.c:9
(gdb) up
#1  0x0000557097e0816b in main (argc=1, argv=0x7ffcffc4ba18) at simple.c:9
9           int ret = myfunc(i);
(gdb) p argc
$1 = 1

So after running bt, we immediately understand where the code was when it crashed, which is sometimes good enough to solve the bug.

As you can see from the example, you are now able to inspect program memory at the time of crash to try and determine the cause of failure, the process virtual memory is entirely contained in the core file.

Tested in Ubuntu 16.04 and 22.04 amd64.

You can also run the program through GDB directly

If the problem is easy to reproduce (i.e. crashes fast and deterministically), and you can easily control the command line (i.e. not a program that is called by another program which you don't want/can't modify) then the best approach is to just run the program through GDB:

gdb -ex run simple.out

and when the signal is received, GDB by default breaks at the signal cause, and we would be left in a situation that looks exactly as when we used the core file.

Direct Binutils analysis

Let's try to observe the contents of the core file without GDB to understand it a bit better. Because we can.

Let's create a program that prints its some of its own memory addresses so we can correlate things:

main.c

#include <stddef.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int myfunc(int i) {
    *(int*)(NULL) = i; /* line 7 */
    return i - 1;
}

int main(int argc, char **argv) {
    /* Setup some memory. */
    char data_ptr[] = "string in data segment";
    char *mmap_ptr;
    char *text_ptr = "string in text segment";
    (void)argv;
    mmap_ptr = (char *)malloc(sizeof(data_ptr) + 1);
    strcpy(mmap_ptr, data_ptr);
    mmap_ptr[10] = 'm';
    mmap_ptr[11] = 'm';
    mmap_ptr[12] = 'a';
    mmap_ptr[13] = 'p';
    printf("text addr: %p\n", text_ptr);
    printf("data addr: %p\n", data_ptr);
    printf("mmap addr: %p\n", mmap_ptr);

    /* Call a function to prepare a stack trace. */
    return myfunc(argc);
}

Program output:

text addr: 0x4007d4
data addr: 0x7ffec6739220
mmap addr: 0x1612010
Segmentation fault (core dumped)

First:

file core

tells us that the core file is actually an ELF file:

core: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from './main.out'

which is why we are able to inspect it more directly with usual binutils tools.

A quick look at the ELF standard shows that there is actually an ELF type dedicated to it:

Elf32_Ehd.e_type == ET_CORE

Further format information can be found at:

man 5 core

Then:

readelf -Wa core

gives some hints about the file structure. Memory appears to be contained in regular program headers:

Program Headers:
  Type           Offset   VirtAddr           PhysAddr           FileSiz  MemSiz   Flg Align
  NOTE           0x000468 0x0000000000000000 0x0000000000000000 0x000b9c 0x000000     0
  LOAD           0x002000 0x0000000000400000 0x0000000000000000 0x001000 0x001000 R E 0x1000
  LOAD           0x003000 0x0000000000600000 0x0000000000000000 0x001000 0x001000 R   0x1000
  LOAD           0x004000 0x0000000000601000 0x0000000000000000 0x001000 0x001000 RW  0x1000

and there is some more metadata present in a notes area, notably prstatus contains the PC:

Displaying notes found at file offset 0x00000468 with length 0x00000b9c:
  Owner                 Data size       Description
  CORE                 0x00000150       NT_PRSTATUS (prstatus structure)
  CORE                 0x00000088       NT_PRPSINFO (prpsinfo structure)
  CORE                 0x00000080       NT_SIGINFO (siginfo_t data)
  CORE                 0x00000130       NT_AUXV (auxiliary vector)
  CORE                 0x00000246       NT_FILE (mapped files)
    Page size: 4096
                 Start                 End         Page Offset
    0x0000000000400000  0x0000000000401000  0x0000000000000000
        /home/ciro/test/main.out
    0x0000000000600000  0x0000000000601000  0x0000000000000000
        /home/ciro/test/main.out
    0x0000000000601000  0x0000000000602000  0x0000000000000001
        /home/ciro/test/main.out
    0x00007f8d939ee000  0x00007f8d93bae000  0x0000000000000000
        /lib/x86_64-linux-gnu/libc-2.23.so
    0x00007f8d93bae000  0x00007f8d93dae000  0x00000000000001c0
        /lib/x86_64-linux-gnu/libc-2.23.so
    0x00007f8d93dae000  0x00007f8d93db2000  0x00000000000001c0
        /lib/x86_64-linux-gnu/libc-2.23.so
    0x00007f8d93db2000  0x00007f8d93db4000  0x00000000000001c4
        /lib/x86_64-linux-gnu/libc-2.23.so
    0x00007f8d93db8000  0x00007f8d93dde000  0x0000000000000000
        /lib/x86_64-linux-gnu/ld-2.23.so
    0x00007f8d93fdd000  0x00007f8d93fde000  0x0000000000000025
        /lib/x86_64-linux-gnu/ld-2.23.so
    0x00007f8d93fde000  0x00007f8d93fdf000  0x0000000000000026
        /lib/x86_64-linux-gnu/ld-2.23.so
  CORE                 0x00000200       NT_FPREGSET (floating point registers)
  LINUX                0x00000340       NT_X86_XSTATE (x86 XSAVE extended state)

objdump can easily dump all memory with:

objdump -s core

which contains:

Contents of section load1:

 4007d0 01000200 73747269 6e672069 6e207465  ....string in te
 4007e0 78742073 65676d65 6e740074 65787420  xt segment.text 

Contents of section load15:

 7ffec6739220 73747269 6e672069 6e206461 74612073  string in data s
 7ffec6739230 65676d65 6e740000 00a8677b 9c6778cd  egment....g{.gx.

Contents of section load4:

 1612010 73747269 6e672069 6e206d6d 61702073  string in mmap s
 1612020 65676d65 6e740000 11040000 00000000  egment..........

which matches exactly with the stdout value in our run.

Tested in Ubuntu 16.04 amd64, GCC 6.4.0, binutils 2.26.1.

Mozilla rr reverse debugging as the ultimate "core file"

Core files allow you to inspect the stack at break.

But in general what you really need to do is to go back in time to further decide the root failure cause.

The amazing Mozilla rr allows you to do that, at the cost of a larger trace file, and a slight performance hit.

Example at: https://stackoverflow.com/questions/1470434/how-does-reverse-debugging-work/53063242#53063242

See also

• https://stackoverflow.com/questions/8305866/how-do-i-analyze-a-programs-core-dump-file-with-gdb-when-it-has-command-line-pa

Answer 2

gdb is the GNU debugger which can be used to examine the core file. BTW bt (backtrace) is a useful gdb command to examine the program call stack.

gdb binary-file core-file

Answer 3

If prefer to use command line tool, then you can use gdb :

gdb <program> <core file>

or

gdb <program> -c <core file>

If you like gui, then install ddd, and from there open the program to debug and the core file.

Answer 4

When you compile the program use -g option

gcc -g program.c

If core file is created then you can debug using gdb whithout using -g option debug flags wont be enabled.

Answer 5

#-------------------------------------------------------------------------
#!/usr/bin/ksh
# -------------------------------------------------------------------------

_OUTFILE=XXXX-XXXX-Audit-`date +"%Y%m%d%H%M"`.log
>$_OUTFILE
MAILLIST=""
COREPATH=$PKMS/logs/cores
MARKER=$COREPATH/marker

function Parse
{
   while getopts :p:u:s:l: name
      do
    case $name in
        p) PKMS="$OPTARG" ;;       # $PKMS
        u) DBUSER="$OPTARG" ;;     # $DBUSER 
        s) DBPSWD="$OPTARG" ;;     # $DBPSWD
        l) DBLOCN="$OPTARG" ;;     # $DBLOC 
        *) Usage ;;                     # display usage and exit
       esac
      done
   if [[ -z "${PKMS}"  || -z "${DBUSER}" || -z "${DBPSWD}" || -z "${DBLOCN}" ]] 
   then
    echo $Usage
    exit -1
   fi
}


function getCoreDumps
{
   COREFILES=$COREPATH/newcores.txt
   STACKS=$COREPATH/stacks.txt
   DATE=$(date +%y%m%d%H%M%S)
   >$COREFILES
   >$STACKS
   umask 002

   find $COREPATH -type f -newer $MARKER -name "core" > $COREFILES
   find $COREPATH -type f -newer $MARKER -name "core.?" >> $COREFILES

   rm $STACKS 2>/dev/null

   for i in $(<$COREFILES)
   do
        mv $i $i.$DATE
        chmod g+r,g+w $i.$DATE
        #echo "Coredump recently found at" `date` '\n'>> $STACKS
        echo $i.$DATE >> $STACKS
    #echo >> $STACKS
   done

   NL=$(wc -l $COREFILES  | awk '{ print $1 }')
   if [ "$NL" -gt 0 ]
   then
    echo "New CORE files found:" >> $_OUTFILE
    echo "--- ---- ----- ------" >> $_OUTFILE
    cat $STACKS >> $_OUTFILE
   else
    echo "No new CORE files found" >> $_OUTFILE
    echo "-- --- ---- ----- -----" >> $_OUTFILE
   fi

}



#/usr/bin/clear

echo "\t\t\t\t---------------------------------\t" >> $_OUTFILE
echo "\t\t\t\t
echo "\t\t\t\t---------------------------------\t" >> $_OUTFILE

date "+                             %d/%m/%Y %H:%M:%S"  >> $_OUTFILE

echo "===================" >> $_OUTFILE
echo " APPICATION MACHINES" >> $_OUTFILE
echo "===================" >> $_OUTFILE
echo >> $_OUTFILE
echo >> $_OUTFILE



getCoreDumps
echo >> $_OUTFILE
echo >> $_OUTFILE



echo "===================" >> $_OUTFILE
echo "XXXX APP DataBase Info" >> $_OUTFILE
echo "===================" >> $_OUTFILE

echo >> $_OUTFILE
getAPPDBInfo
echo >> $_OUTFILE
echo >> $_OUTFILE

MAILDATE=$(date +%d/%m/%Y)


mailx -s "XXXX Monitor Log for $PKMS Environment - Dated $MAILDATE" $MAILLIST < $_OUTFILE

touch $MARKER
rm /tmp/XXXXtempOUTFILE
exit 0