0

My understanding is that the Linux kernel allocates a stack to every process before it starts, and this initial size is configurable. A process can PUSH data to the stack directly via CPU instructions, i.e. without involving the OS in each such step.

At some point, however, the PUSH operation may exceed the allocated stack size, and a variety of things can happen depending on whether or not the Kernel can allocate a larger stack size.

My question is: What happens right after that initial PUSH? How does the system detect the "stack overflow", and that it needs to grow the stack until it eventually resumes execution of the process?

We can focus on the x86 arch if helpful.

I'd imagine we have the following parties involved:

  • CPU
  • MMU
  • Kernel
  • Process

For example, who detects the "stack overflow, is it the CPU? the MMU? Does that result in issuing a HW interrupt? Something else? and if so, who handles what here? (until the stack is finally expanded, and the process can resume its next CPU instruction?)

Improve this question

1 Answer 1

Reset to default
1

How does stack allocation work in Linux?

Is a stack overflow detected by hardware or software?

The MMU will leave some padding filled with an invalid memory address after the expected size of the stack, if you try to grow the stack too much you will end up de-referencing and invalid memory address and the MMU will throw a seg fault.

You can also use canari values in the stack but I won't go into.

Usually compilers will perform static analysis on the code and try to find the size that the stack requires to prevent stack resize. -fstack-usage gives you insight into what gcc does

The C standard defines that when a C stack needs to grow it will implicitly call an mremap which means that the Linux kernel handles all of that by itself.

An overly simplified timeline would look like this:

PUSH -> MMU seg-faults -> Linux detects it -> Linux remaps memory -> the value is inserted -> the program continues

I would highly encourage you no to take this as a definitive answer and to go read more on the subject as I am not an expert.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .