How does a debugger work in Linux?

Question

How does a debugger work in Linux? How does it gets 'attached' to an already running executable or process. I understand that compiler translates code to machine language, but then how does debugger 'know' what it is being attached to?

Answer 1

There is a system call named ptrace. It takes 4 parameters: the operation, the PID of the target process, an address in the target process memory, and a data pointer. The way the last 2 parameters are used is dependent on the operation.

For example you can attach/detach your debugger to a process:

ptrace(PTRACE_ATTACH, pid, 0, 0);
...
ptrace(PTRACE_DETACH, pid, 0, 0);

Single step execution:

ptrace(PTRACE_ATTACH, pid, 0, 0);
int status;
waitpid(pid, &status, WSTOPPED);
while (...) {
    ptrace(PTRACE_SINGLESTEP, pid, 0, 0);
    // give the user a chance to do something
}
ptrace(PTRACE_DETACH, pid, 0, 0);

You can also read/write the memory of the target process with PTRACE_PEEKDATA and PTRACE_POKEDATA. If you want to see a real example check out gdb.

Answer 2

but then how does debugger 'know' what it is being attached to?

from man ptrace

The ptrace() system call provides a means by which one process (the "tracer") may observe and control the execution of another process (the "tracee"), and examine and change the tracee's memory and registers. It is primarily used to implement breakpoint debugging and system call tracing.

How Debuggers Work Part 1 of 3 is a good reference for using ptrace, the debugging system call.

Very briefly, because gdb is a complicated beast. In the gdb source code as of today, the interesting information regarding gdb's use of ptrace is found in target.h and inf-ptrace.c and can easily be chased down from there by the ambitious. You'll see that gdb is using ptrace, usually without much ambiguity at all. Refer to the target_ops structure's components and find them in inf-ptrace.c.

From man ptrace

>PTRACE_ATTACH
>       Attach to the process specified in pid, making it a tracee of the calling process.  The
>       tracee is sent a SIGSTOP, but will not necessarily have stopped by  the  completion  of
>       this  call;  use  waitpid(2)  to  wait  for the tracee to stop.  See the "Attaching and
>       detaching" subsection for additional information.  (addr and data are ignored.)

So ptrace is used, but how does it know where anything is? How does GDB get the information it needs to set breakpoints and read data? Without DWARF or some lesser debugging format, gdb doesn't really have a clue what it's doing or where to find anything. You can still find them with gdb by searching manually, because the information is still there in the image of the program, but gdb doesn't have a clue.

Below is from GDB User Manual-Stallman. GDB, as with likely all debuggers, uses a debugging format, typically DWARF on a reasonable operating system, for symbol information regarding the functions and variables of the program.

In order to debug a program effectively, you need to generate debugging information when you compile it. This debugging information is stored in the object file; it describes the data type of each variable or function and the correspondence between source line numbers and addresses in the executable code. .... You will have the best debugging experience if you use the latest version of the DWARF debugging format that your compiler supports.

Information on GDB, DWARF and PTRACE are all exceedingly easy to find. And if you're fancy is symbols, info on ELF files are easy to find too.