How does a debugger work in Linux?

Question

How does a debugger work in Linux? How does it gets 'attached' to an already running executable or process. I understand that compiler translates code to machine language, but then how does debugger 'know' what it is being attached to?

Answer 1

There is a system call named ptrace. It takes 4 parameters: the operation, the PID of the target process, an address in the target process memory, and a data pointer. The way the last 2 parameters are used is dependent on the operation.

For example you can attach/detach your debugger to a process:

ptrace(PTRACE_ATTACH, pid, 0, 0);
...
ptrace(PTRACE_DETACH, pid, 0, 0);

Single step execution:

ptrace(PTRACE_ATTACH, pid, 0, 0);
int status;
waitpid(pid, &status, WSTOPPED);
while (...) {
    ptrace(PTRACE_SINGLESTEP, pid, 0, 0);
    // give the user a chance to do something
}
ptrace(PTRACE_DETACH, pid, 0, 0);

You can also read/write the memory of the target process with PTRACE_PEEKDATA and PTRACE_POKEDATA. If you want to see a real example check out gdb.