How to view core files for debugging purposes in Linux?

Question

I want to view the contents of a core file while debugging a program. How can I view the contents of a core file?

Answer 1

objdump + gdb minimal runnable example

TLDR:

• objdump -s core to dump memory

• GDB to find failing line, previously mentioned at: How to view core files for debugging purposes in Linux?

Now for a the full educational test setup:

main.c

#include <stddef.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int myfunc(int i) {
    *(int*)(NULL) = i; /* line 7 */
    return i - 1;
}

int main(int argc, char **argv) {
    /* Setup some memory. */
    char data_ptr[] = "string in data segment";
    char *mmap_ptr;
    char *text_ptr = "string in text segment";
    (void)argv;
    mmap_ptr = (char *)malloc(sizeof(data_ptr) + 1);
    strcpy(mmap_ptr, data_ptr);
    mmap_ptr[10] = 'm';
    mmap_ptr[11] = 'm';
    mmap_ptr[12] = 'a';
    mmap_ptr[13] = 'p';
    printf("text addr: %p\n", text_ptr);
    printf("data addr: %p\n", data_ptr);
    printf("mmap addr: %p\n", mmap_ptr);

    /* Call a function to prepare a stack trace. */
    return myfunc(argc);
}

Compile, and run to generate core:

gcc -ggdb3 -std=c99 -Wall -Wextra -pedantic -o main.out main.c
ulimit -c unlimited
rm -f core
./main.out

Output:

text addr: 0x4007d4
data addr: 0x7ffec6739220
mmap addr: 0x1612010
Segmentation fault (core dumped)

GDB points us to the exact line where the segfault happened, which is what most users want while debugging:

gdb -q -nh main.out core

then:

Reading symbols from main.out...done.
[New LWP 27479]
Core was generated by `./main.out'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000400635 in myfunc (i=1) at main.c:7
7           *(int*)(NULL) = i;
(gdb) bt
#0  0x0000000000400635 in myfunc (i=1) at main.c:7
#1  0x000000000040072b in main (argc=1, argv=0x7ffec6739328) at main.c:28

which points us directly to the buggy line 7.

Binutils analysis

First:

file core

tells us that the core file is actually an ELF file:

core: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from './main.out'

which is why we are able to inspect it more directly with usual binutils tools.

A quick look at the ELF standard shows that there is actually an ELF type dedicated to it:

Elf32_Ehd.e_type == ET_CORE

Further format information can be found at:

man 5 core

Then:

readelf -Wa core

gives some hints about the file structure. Memory appears to be contained in regular program headers:

Program Headers:
  Type           Offset   VirtAddr           PhysAddr           FileSiz  MemSiz   Flg Align
  NOTE           0x000468 0x0000000000000000 0x0000000000000000 0x000b9c 0x000000     0
  LOAD           0x002000 0x0000000000400000 0x0000000000000000 0x001000 0x001000 R E 0x1000
  LOAD           0x003000 0x0000000000600000 0x0000000000000000 0x001000 0x001000 R   0x1000
  LOAD           0x004000 0x0000000000601000 0x0000000000000000 0x001000 0x001000 RW  0x1000

and there is some more metadata present in a notes area, notably prstatus contains the PC:

Displaying notes found at file offset 0x00000468 with length 0x00000b9c:
  Owner                 Data size       Description
  CORE                 0x00000150       NT_PRSTATUS (prstatus structure)
  CORE                 0x00000088       NT_PRPSINFO (prpsinfo structure)
  CORE                 0x00000080       NT_SIGINFO (siginfo_t data)
  CORE                 0x00000130       NT_AUXV (auxiliary vector)
  CORE                 0x00000246       NT_FILE (mapped files)
    Page size: 4096
                 Start                 End         Page Offset
    0x0000000000400000  0x0000000000401000  0x0000000000000000
        /home/ciro/test/main.out
    0x0000000000600000  0x0000000000601000  0x0000000000000000
        /home/ciro/test/main.out
    0x0000000000601000  0x0000000000602000  0x0000000000000001
        /home/ciro/test/main.out
    0x00007f8d939ee000  0x00007f8d93bae000  0x0000000000000000
        /lib/x86_64-linux-gnu/libc-2.23.so
    0x00007f8d93bae000  0x00007f8d93dae000  0x00000000000001c0
        /lib/x86_64-linux-gnu/libc-2.23.so
    0x00007f8d93dae000  0x00007f8d93db2000  0x00000000000001c0
        /lib/x86_64-linux-gnu/libc-2.23.so
    0x00007f8d93db2000  0x00007f8d93db4000  0x00000000000001c4
        /lib/x86_64-linux-gnu/libc-2.23.so
    0x00007f8d93db8000  0x00007f8d93dde000  0x0000000000000000
        /lib/x86_64-linux-gnu/ld-2.23.so
    0x00007f8d93fdd000  0x00007f8d93fde000  0x0000000000000025
        /lib/x86_64-linux-gnu/ld-2.23.so
    0x00007f8d93fde000  0x00007f8d93fdf000  0x0000000000000026
        /lib/x86_64-linux-gnu/ld-2.23.so
  CORE                 0x00000200       NT_FPREGSET (floating point registers)
  LINUX                0x00000340       NT_X86_XSTATE (x86 XSAVE extended state)

objdump can easily dump all memory with:

objdump -s core

which contains:

Contents of section load1:

 4007d0 01000200 73747269 6e672069 6e207465  ....string in te
 4007e0 78742073 65676d65 6e740074 65787420  xt segment.text 

Contents of section load15:

 7ffec6739220 73747269 6e672069 6e206461 74612073  string in data s
 7ffec6739230 65676d65 6e740000 00a8677b 9c6778cd  egment....g{.gx.

Contents of section load4:

 1612010 73747269 6e672069 6e206d6d 61702073  string in mmap s
 1612020 65676d65 6e740000 11040000 00000000  egment..........

which matches exactly with the stdout value in our run.

Tested in Ubuntu 16.04 amd64, GCC 6.4.0, binutils 2.26.1.

Answer 2

gdb is the GNU debugger which can be used to examine the core file. BTW bt (backtrace) is a useful gdb command to examine the program call stack.

gdb binary-file core-file

Answer 3

If prefer to use command line tool, then you can use gdb :

gdb <program> <core file>

or

gdb <program> -c <core file>

If you like gui, then install ddd, and from there open the program to debug and the core file.

Answer 4

When you compile the program use -g option

gcc -g program.c

If core file is created then you can debug using gdb whithout using -g option debug flags wont be enabled.

Answer 5

#-------------------------------------------------------------------------
#!/usr/bin/ksh
# -------------------------------------------------------------------------

_OUTFILE=XXXX-XXXX-Audit-`date +"%Y%m%d%H%M"`.log
>$_OUTFILE
MAILLIST=""
COREPATH=$PKMS/logs/cores
MARKER=$COREPATH/marker

function Parse
{
   while getopts :p:u:s:l: name
      do
    case $name in
        p) PKMS="$OPTARG" ;;       # $PKMS
        u) DBUSER="$OPTARG" ;;     # $DBUSER 
        s) DBPSWD="$OPTARG" ;;     # $DBPSWD
        l) DBLOCN="$OPTARG" ;;     # $DBLOC 
        *) Usage ;;                     # display usage and exit
       esac
      done
   if [[ -z "${PKMS}"  || -z "${DBUSER}" || -z "${DBPSWD}" || -z "${DBLOCN}" ]] 
   then
    echo $Usage
    exit -1
   fi
}


function getCoreDumps
{
   COREFILES=$COREPATH/newcores.txt
   STACKS=$COREPATH/stacks.txt
   DATE=$(date +%y%m%d%H%M%S)
   >$COREFILES
   >$STACKS
   umask 002

   find $COREPATH -type f -newer $MARKER -name "core" > $COREFILES
   find $COREPATH -type f -newer $MARKER -name "core.?" >> $COREFILES

   rm $STACKS 2>/dev/null

   for i in $(<$COREFILES)
   do
        mv $i $i.$DATE
        chmod g+r,g+w $i.$DATE
        #echo "Coredump recently found at" `date` '\n'>> $STACKS
        echo $i.$DATE >> $STACKS
    #echo >> $STACKS
   done

   NL=$(wc -l $COREFILES  | awk '{ print $1 }')
   if [ "$NL" -gt 0 ]
   then
    echo "New CORE files found:" >> $_OUTFILE
    echo "--- ---- ----- ------" >> $_OUTFILE
    cat $STACKS >> $_OUTFILE
   else
    echo "No new CORE files found" >> $_OUTFILE
    echo "-- --- ---- ----- -----" >> $_OUTFILE
   fi

}



#/usr/bin/clear

echo "\t\t\t\t---------------------------------\t" >> $_OUTFILE
echo "\t\t\t\t
echo "\t\t\t\t---------------------------------\t" >> $_OUTFILE

date "+                             %d/%m/%Y %H:%M:%S"  >> $_OUTFILE

echo "===================" >> $_OUTFILE
echo " APPICATION MACHINES" >> $_OUTFILE
echo "===================" >> $_OUTFILE
echo >> $_OUTFILE
echo >> $_OUTFILE



getCoreDumps
echo >> $_OUTFILE
echo >> $_OUTFILE



echo "===================" >> $_OUTFILE
echo "XXXX APP DataBase Info" >> $_OUTFILE
echo "===================" >> $_OUTFILE

echo >> $_OUTFILE
getAPPDBInfo
echo >> $_OUTFILE
echo >> $_OUTFILE

MAILDATE=$(date +%d/%m/%Y)


mailx -s "XXXX Monitor Log for $PKMS Environment - Dated $MAILDATE" $MAILLIST < $_OUTFILE

touch $MARKER
rm /tmp/XXXXtempOUTFILE
exit 0