How does a debugger work in Linux? How does it gets 'attached' to an already running executable or process. I understand that compiler translates code to machine language, but then how does debugger 'know' what it is being attached to?
There is a system call named ptrace. It takes 4 parameters: the operation, the PID of the target process, an address in the target process memory, and a data pointer. The way the last 2 parameters are used is dependent on the operation.
For example you can attach/detach your debugger to a process:
Single step execution:
You can also read/write the memory of the target process with PTRACE_PEEKDATA and PTRACE_POKEDATA. If you want to see a real example check out gdb.