4

I want to run two machine:

  1. debugged machine: compiled kernel with kgdb option.
  2. debugger machine: for running gdb

How can I debug machine 1 from machine 2?

  • What have you done this far? Have you setup your machine to be debugged with a serial console which is reachable from your debugger machine? – Sami Laine Dec 15 '13 at 10:55
  • @SamiLaine Yes I've. I can debug my machine that need to be debugged from serial console in host machine. but I need run two virtual machine and debug one of them through another. – Milad Khajavi Dec 15 '13 at 10:58
  • I think you should mention that you're using virtual machines in the question, that helps a lot. The actual solution depends partially on which virtualization technology (e.g. Xen or VMware) you're using. – Sami Laine Dec 15 '13 at 11:06
  • @SamiLaine I think I need to convert serial console to socket, in my host machine so I can debug one vm machine from another vm machine. – Milad Khajavi Dec 15 '13 at 11:10
  • @SamiLaine I'm using this tutorial: fotisl.com/blog/?p=25 I've compiled kernel by kgdb option (vm machine A) and now I can connect the gdb by serial console to machine A. but now I need to run new vm machine B, and then through machin B, debug machine A. – Milad Khajavi Dec 15 '13 at 11:17
2

gdbserver

Sounds like you're looking for gdbserver.

gdbserver is a control program for Unix-like systems, which allows you to connect your program with a remote GDB via target remote---but without linking in the usual debugging stub.

Example

On the target machine

You need to have a copy of the program you want to debug. gdbserver does not need your program's symbol table, so you can strip the program if necessary to save space. GDB on the host system does all the symbol handling.

target$ gdbserver host:2345 emacs foo.txt

NOTE: You can also attach to running processes like so:

target$ gdbserver comm --attach pid

One the GDB host machine

You need an unstripped copy of your program, since GDB needs symbols and debugging information. Start up GDB as usual, using the name of the local copy of your program as the first argument. (You may also need the --baud' option if the serial line is running at anything other than 9600bps.) After that, use target remote to establish communications with gdbserver. Its argument is either a device name (usually a serial device, like/dev/ttyb'), or a TCP port descriptor in the form host:PORT. For example:

   (gdb) target remote the-target:2345

GDB stub

There is another method discussed in the manuals, called "remote stub". The official manuals are located here, GDB Documentation, on the gnu.org website. Looking through the GDB Users Manual, section 20.5, Implementing a Remote Stub, explains how to use this feature instead of gdbserver.

This method is described as follows in the docs:

The next step is to arrange for your program to use a serial port to communicate with the machine where gdb is running (the host machine). In general terms, the scheme looks like this:

So you might be able to setup a serial port on both the VM host and the guest and debug the guest's kernel using this method.

References

1

KGDB + QEMU step-by-step

My QEMU + Buildroot example is a good way to get a taste of it without real hardware: https://github.com/cirosantilli/linux-kernel-module-cheat/tree/1969cd6f8d30dace81d9848c6bacbb8bad9dacd8#kgdb

Pros and cons vs other methods:

  • advantage vs QEMU:
    • you often don't have software emulation for your device as hardware vendors don't like to release accurate software models for their devices
    • real hardware way faster than QEMU
  • advantage vs JTAG: no need for extra JTAG hardware, easier to setup
  • disadvantages vs QEMU and JTAG: less visibility and more intrusive. KGDB relies on the certain parts of the kernel working to be able to communicate with the host. So e.g. it breaks down in panic, you can't view the boot sequence.

The main steps are:

  1. Compile the kernel with:

    CONFIG_DEBUG_KERNEL=y
    CONFIG_DEBUG_INFO=y
    
    CONFIG_CONSOLE_POLL=y
    CONFIG_KDB_CONTINUE_CATASTROPHIC=0
    CONFIG_KDB_DEFAULT_ENABLE=0x1
    CONFIG_KDB_KEYBOARD=y
    CONFIG_KGDB=y
    CONFIG_KGDB_KDB=y
    CONFIG_KGDB_LOW_LEVEL_TRAP=y
    CONFIG_KGDB_SERIAL_CONSOLE=y
    CONFIG_KGDB_TESTS=y
    CONFIG_KGDB_TESTS_ON_BOOT=n
    CONFIG_MAGIC_SYSRQ=y
    CONFIG_MAGIC_SYSRQ_DEFAULT_ENABLE=0x1
    CONFIG_SERIAL_KGDB_NMI=n
    

    Most of those are not mandatory, but this is what I've tested.

  2. Add to your QEMU command:

    -append 'kgdbwait kgdboc=ttyS0,115200' \
    -serial tcp::1234,server,nowait
    
  3. Run GDB with from the root of the Linux kernel source tree with:

    gdb -ex 'file vmlinux' -ex 'target remote localhost:1234'
    
  4. In GDB:

    (gdb) c
    

    and the boot should finish.

  5. In QEMU:

    echo g > /proc/sysrq-trigger
    

    And GDB should break.

  6. Now we are done, you can use GDB as usual:

    b sys_write
    c
    

Tested in Ubuntu 14.04.

KGDB + Raspberry Pi

The exact same setup as above almost worked on a Raspberry Pi 2, Raspbian Jessie 2016-05-27.

You just have to learn to do the QEMU steps on the Pi, which are easily Googlable:

  • add the configuration options and recompile the kernel as explained at https://www.raspberrypi.org/documentation/linux/kernel/building.md There were unfortunately missing options on the default kernel build, notably no debug symbols, so the recompile is needed.

  • edit cmdline.txt of the boot partition and add:

    kgdbwait kgdboc=ttyAMA0,115200
    
  • connect gdb to the serial with:

    gdb -ex 'file vmlinux' -ex 'target remote /dev/ttyUSB0'
    

    If you are not familiar with the serial, check out this: https://www.youtube.com/watch?v=da5Q7xL_OTo All you need is a cheap adapter like this one. Make sure you can get a shell through the serial to ensure that it is working before trying out KGDB.

  • do:

    echo g | sudo tee /proc/sysrq-trigger
    

    from inside an SSH session, since the serial is already taken by GDB.

With this setup, I was able to put a breakpoint in sys_write, pause program execution, list source and continue.

However, sometimes when I did next in sys_write GDB just hung and printed this error message several times:

Ignoring packet error, continuing...

so I'm not sure if something is wrong with my setup, or if this is expected because of what some background process is doing in the more complex Raspbian image.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.