2

I'm trying to obtain a user stack trace in ftrace, but for some reason, only my user program cannot resolve the addresses to symbols. First, the OS I'm using is:

$ uname -a
Linux mypc 2.6.38-16-generic #67-Ubuntu SMP Thu Sep 6 18:00:43 UTC 2012 i686 i686 i386 GNU/Linux

The (trivial) user-space program I'm using, wtest.c, is included below. I compile it with:

gcc -g -pg -O0 wtest.c -o wtest

... hoping that -g and -pg will include enough debugging symbol information.

Then, I run the wtest program under ftrace. Note that the I want to obtain the user space stacktrace as a sort of a guide where in my user space program I am currently, while I otherwise run the function tracer. The Linux/Documentation/trace/ftrace.txt isn't explicit about this, but in order to have a stack trace, there must be something to trigger it - and just the function (or function_graph) tracer, on its own, will not do any such triggering. So I choose to trigger anytime a system call is detected via events/raw_syscalls. Furthermore, the stack trace will print addresses only, unless sym-userobj is specified. So, I run this script to do the tracing:

sudo bash -c '
KDBGPATH="/sys/kernel/debug/tracing"
echo 1 > $KDBGPATH/events/raw_syscalls/enable
echo userstacktrace > $KDBGPATH/trace_options
echo sym-userobj > $KDBGPATH/trace_options
echo function > $KDBGPATH/current_tracer ;
echo 0 > $KDBGPATH/tracing_on
echo > $KDBGPATH/trace
echo 1 > $KDBGPATH/tracing_on ; /media/disk/wtest ; echo 0 > $KDBGPATH/tracing_on
cat $KDBGPATH/trace > wtest.ftrace
'

However, the trace ends up containing something like this:

...
           wtest-16108 [000] 16246.413238: arch_flush_lazy_mmu_mode <-__kunmap_atomic
            Xorg-1136  [001] 16246.413239: syscall_trace_leave <-syscall_exit_work
           wtest-16108 [000] 16246.413240: up_read <-do_page_fault
            Xorg-1136  [001] 16246.413241: sys_exit: NR 54 = 0
            Xorg-1136  [001] 16246.413242: <user stack trace>
 =>  <008f9416>
 => /lib/i386-linux-gnu/libdrm_intel.so.1.0.0[+0x4f49]
 => /lib/i386-linux-gnu/libdrm_intel.so.1.0.0[+0x5cc3]
 => /lib/i386-linux-gnu/libdrm_intel.so.1.0.0[+0x13a3]
 => /usr/lib/xorg/modules/drivers/intel_drv.so[+0xec58]
 => /usr/lib/xorg/modules/drivers/intel_drv.so[+0x2957a]
 => /usr/lib/xorg/modules/drivers/intel_drv.so[+0x2d403]
 => /usr/bin/Xorg[+0xdefc8]
            Xorg-1136  [001] 16246.413242: save_stack_trace_user <-ftrace_trace_userstack
            Xorg-1136  [001] 16246.413242: __copy_from_user_ll_nozero <-save_stack_trace_user
            Xorg-1136  [001] 16246.413242: __copy_from_user_ll_nozero <-save_stack_trace_user
            Xorg-1136  [001] 16246.413242: __copy_from_user_ll_nozero <-save_stack_trace_user
            Xorg-1136  [001] 16246.413242: __copy_from_user_ll_nozero <-save_stack_trace_user
            Xorg-1136  [001] 16246.413242: __copy_from_user_ll_nozero <-save_stack_trace_user
            Xorg-1136  [001] 16246.413242: __copy_from_user_ll_nozero <-save_stack_trace_user
            Xorg-1136  [001] 16246.413242: __copy_from_user_ll_nozero <-save_stack_trace_user
           wtest-16108 [000] 16246.413242: syscall_trace_leave <-syscall_exit_work
           wtest-16108 [000] 16246.413243: sys_exit: NR 120 = 0
           wtest-16108 [000] 16246.413245: <user stack trace>
 =>  <00222416>
 =>  <08086a69>
 =>  <080753e0>
 =>  <080783b5>
 =>  <08073c09>
 =>  <08077c93>
 =>  <08078374>
 =>  <08073c09>
           wtest-16108 [000] 16246.413245: save_stack_trace_user <-ftrace_trace_userstack
           wtest-16108 [000] 16246.413245: __copy_from_user_ll_nozero <-save_stack_trace_user
           wtest-16108 [000] 16246.413245: __copy_from_user_ll_nozero <-save_stack_trace_user
           wtest-16108 [000] 16246.413245: __copy_from_user_ll_nozero <-save_stack_trace_user
           wtest-16108 [000] 16246.413245: __copy_from_user_ll_nozero <-save_stack_trace_user
           wtest-16108 [000] 16246.413245: __copy_from_user_ll_nozero <-save_stack_trace_user
           wtest-16108 [000] 16246.413245: __copy_from_user_ll_nozero <-save_stack_trace_user
           wtest-16108 [000] 16246.413245: __copy_from_user_ll_nozero <-save_stack_trace_user
           wtest-16108 [000] 16246.413260: down_read_trylock <-do_page_fault
           wtest-16108 [000] 16246.413262: _cond_resched <-do_page_fault
           wtest-16108 [000] 16246.413263: find_vma <-do_page_fault
           wtest-16108 [000] 16246.413265: handle_mm_fault <-do_page_fault
           wtest-16108 [000] 16246.413266: __kmap_atomic <-handle_mm_fault
...

In other words - Xorg has its user stack trace resolved fine (at least in the .so libraries) - and pretty much all other processes ending up in the ftrace log (like bash, etc); however, the user stack trace for wtest is printed with addresses only ?! I explicitly called the executable through its absolute path, but that doesn't seem to make much difference.

Is there a way I could compile my wtest program, so its user stack trace gets its symbols resolved - or does the problem lie elsewhere? (I have tried -ggdb and -gdwarf-2 instead of -g, but those don't really help here...)

Here is the code for wtest.c:

#include <stdio.h>
#include <fcntl.h>  // O_CREAT, O_WRONLY, S_IRUSR

int main(void) {
  char filename[] = "/tmp/wtest.txt";
  char buffer[] = "abcd";
  int fd;
  mode_t perms = S_IRUSR|S_IWUSR|S_IRGRP|S_IWGRP|S_IROTH|S_IWOTH;

  fd = open(filename, O_RDWR|O_CREAT, perms);
  write(fd,buffer,4);
  close(fd);

  return 0;
}

1 Answer 1

1

The -g switch is what will enable gcc to compile your code with debugging symbols enabled. If you're not seeing any symbols then the issue is likely a library that you're compiling against, which does not provide a version of the library with symbols enabled.

See this SO Q&A: titled: How Does The Debugging Option -g Change the Binary Executable?. This SO Q&A also makes mention of debugging symbols vs. linker symbols, titled: Why does gcc add symbols to non-debug build?.

If you want the debugging symbols for a particular package/library they're typically called <package name>-dbg or <package name>-dbgsym on Ubuntu. This article discussed it in further details, titled: DebuggingProgramCrash.

3
  • Many thanks for the answer, @slm - could this also refer to the libc library? As I don't see what other library would the program above refer to... Cheers!
    – sdaau
    Jan 22, 2014 at 20:46
  • 1
    @sdaau - Possibly do you happen to know if you have -dbg or -dbgsym packages loaded? You might need those for what you want. See updates.
    – slm
    Jan 22, 2014 at 20:53
  • Thanks for the edits, @slm - I definitely do not have -dbg packages loaded; which is why I find this strange - if they were the problem, then I shouldn't get any resolved traces from Xorg or other processes? And this trace only gives an offset into an executable file, so the debug info (as in, a link to source code file and line number) isn't even needed... Yet, I get stacktraces resolved everywhere, except my program; and the program isn't even linked to anything other than (implicitly) libc?
    – sdaau
    Jan 22, 2014 at 21:02

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .