I'm trying to obtain a user stack trace in ftrace, but for some reason, only my user program cannot resolve the addresses to symbols. First, the OS I'm using is:

$ uname -a
Linux mypc 2.6.38-16-generic #67-Ubuntu SMP Thu Sep 6 18:00:43 UTC 2012 i686 i686 i386 GNU/Linux

The (trivial) user-space program I'm using, wtest.c, is included below. I compile it with:

gcc -g -pg -O0 wtest.c -o wtest

... hoping that -g and -pg will include enough debugging symbol information.

Then, I run the wtest program under ftrace. Note that the I want to obtain the user space stacktrace as a sort of a guide where in my user space program I am currently, while I otherwise run the function tracer. The Linux/Documentation/trace/ftrace.txt isn't explicit about this, but in order to have a stack trace, there must be something to trigger it - and just the function (or function_graph) tracer, on its own, will not do any such triggering. So I choose to trigger anytime a system call is detected via events/raw_syscalls. Furthermore, the stack trace will print addresses only, unless sym-userobj is specified. So, I run this script to do the tracing:

sudo bash -c '
echo 1 > $KDBGPATH/events/raw_syscalls/enable
echo userstacktrace > $KDBGPATH/trace_options
echo sym-userobj > $KDBGPATH/trace_options
echo function > $KDBGPATH/current_tracer ;
echo 0 > $KDBGPATH/tracing_on
echo > $KDBGPATH/trace
echo 1 > $KDBGPATH/tracing_on ; /media/disk/wtest ; echo 0 > $KDBGPATH/tracing_on
cat $KDBGPATH/trace > wtest.ftrace

However, the trace ends up containing something like this:

           wtest-16108 [000] 16246.413238: arch_flush_lazy_mmu_mode <-__kunmap_atomic
            Xorg-1136  [001] 16246.413239: syscall_trace_leave <-syscall_exit_work
           wtest-16108 [000] 16246.413240: up_read <-do_page_fault
            Xorg-1136  [001] 16246.413241: sys_exit: NR 54 = 0
            Xorg-1136  [001] 16246.413242: <user stack trace>
 =>  <008f9416>
 => /lib/i386-linux-gnu/libdrm_intel.so.1.0.0[+0x4f49]
 => /lib/i386-linux-gnu/libdrm_intel.so.1.0.0[+0x5cc3]
 => /lib/i386-linux-gnu/libdrm_intel.so.1.0.0[+0x13a3]
 => /usr/lib/xorg/modules/drivers/intel_drv.so[+0xec58]
 => /usr/lib/xorg/modules/drivers/intel_drv.so[+0x2957a]
 => /usr/lib/xorg/modules/drivers/intel_drv.so[+0x2d403]
 => /usr/bin/Xorg[+0xdefc8]
            Xorg-1136  [001] 16246.413242: save_stack_trace_user <-ftrace_trace_userstack
            Xorg-1136  [001] 16246.413242: __copy_from_user_ll_nozero <-save_stack_trace_user
            Xorg-1136  [001] 16246.413242: __copy_from_user_ll_nozero <-save_stack_trace_user
            Xorg-1136  [001] 16246.413242: __copy_from_user_ll_nozero <-save_stack_trace_user
            Xorg-1136  [001] 16246.413242: __copy_from_user_ll_nozero <-save_stack_trace_user
            Xorg-1136  [001] 16246.413242: __copy_from_user_ll_nozero <-save_stack_trace_user
            Xorg-1136  [001] 16246.413242: __copy_from_user_ll_nozero <-save_stack_trace_user
            Xorg-1136  [001] 16246.413242: __copy_from_user_ll_nozero <-save_stack_trace_user
           wtest-16108 [000] 16246.413242: syscall_trace_leave <-syscall_exit_work
           wtest-16108 [000] 16246.413243: sys_exit: NR 120 = 0
           wtest-16108 [000] 16246.413245: <user stack trace>
 =>  <00222416>
 =>  <08086a69>
 =>  <080753e0>
 =>  <080783b5>
 =>  <08073c09>
 =>  <08077c93>
 =>  <08078374>
 =>  <08073c09>
           wtest-16108 [000] 16246.413245: save_stack_trace_user <-ftrace_trace_userstack
           wtest-16108 [000] 16246.413245: __copy_from_user_ll_nozero <-save_stack_trace_user
           wtest-16108 [000] 16246.413245: __copy_from_user_ll_nozero <-save_stack_trace_user
           wtest-16108 [000] 16246.413245: __copy_from_user_ll_nozero <-save_stack_trace_user
           wtest-16108 [000] 16246.413245: __copy_from_user_ll_nozero <-save_stack_trace_user
           wtest-16108 [000] 16246.413245: __copy_from_user_ll_nozero <-save_stack_trace_user
           wtest-16108 [000] 16246.413245: __copy_from_user_ll_nozero <-save_stack_trace_user
           wtest-16108 [000] 16246.413245: __copy_from_user_ll_nozero <-save_stack_trace_user
           wtest-16108 [000] 16246.413260: down_read_trylock <-do_page_fault
           wtest-16108 [000] 16246.413262: _cond_resched <-do_page_fault
           wtest-16108 [000] 16246.413263: find_vma <-do_page_fault
           wtest-16108 [000] 16246.413265: handle_mm_fault <-do_page_fault
           wtest-16108 [000] 16246.413266: __kmap_atomic <-handle_mm_fault

In other words - Xorg has its user stack trace resolved fine (at least in the .so libraries) - and pretty much all other processes ending up in the ftrace log (like bash, etc); however, the user stack trace for wtest is printed with addresses only ?! I explicitly called the executable through its absolute path, but that doesn't seem to make much difference.

Is there a way I could compile my wtest program, so its user stack trace gets its symbols resolved - or does the problem lie elsewhere? (I have tried -ggdb and -gdwarf-2 instead of -g, but those don't really help here...)

Here is the code for wtest.c:

#include <stdio.h>
#include <fcntl.h>  // O_CREAT, O_WRONLY, S_IRUSR

int main(void) {
  char filename[] = "/tmp/wtest.txt";
  char buffer[] = "abcd";
  int fd;

  fd = open(filename, O_RDWR|O_CREAT, perms);

  return 0;

The -g switch is what will enable gcc to compile your code with debugging symbols enabled. If you're not seeing any symbols then the issue is likely a library that you're compiling against, which does not provide a version of the library with symbols enabled.

See this SO Q&A: titled: How Does The Debugging Option -g Change the Binary Executable?. This SO Q&A also makes mention of debugging symbols vs. linker symbols, titled: Why does gcc add symbols to non-debug build?.

If you want the debugging symbols for a particular package/library they're typically called <package name>-dbg or <package name>-dbgsym on Ubuntu. This article discussed it in further details, titled: DebuggingProgramCrash.

  • Many thanks for the answer, @slm - could this also refer to the libc library? As I don't see what other library would the program above refer to... Cheers!
    – sdaau
    Jan 22 '14 at 20:46
  • 1
    @sdaau - Possibly do you happen to know if you have -dbg or -dbgsym packages loaded? You might need those for what you want. See updates.
    – slm
    Jan 22 '14 at 20:53
  • Thanks for the edits, @slm - I definitely do not have -dbg packages loaded; which is why I find this strange - if they were the problem, then I shouldn't get any resolved traces from Xorg or other processes? And this trace only gives an offset into an executable file, so the debug info (as in, a link to source code file and line number) isn't even needed... Yet, I get stacktraces resolved everywhere, except my program; and the program isn't even linked to anything other than (implicitly) libc?
    – sdaau
    Jan 22 '14 at 21:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.