What do you call the calling convention behind `int 0x80`?

Question

I know there is a syscall convention but what do you call the calling convention that precedes it that you see when you call to int 80 rather than syscall, like this.

mov  rax,4     ; system call number (sys_write)
mov  rbx,1     ; file descriptor (stdout)
mov  rcx,hello ; message to write
mov  rdx,12    ; message length
int  0x80      ; call kernel

I read here that the arguments after rdx are esi, edi, ebp (or for x64 rsi, rdi, rbp), I don't see it documented in Wikipedia's page for calling conventions, but int80h seems to indicate that Windows also uses this convention?

What is this conventioned named. Where in the Linux Kernel source can I see it defined? And, where is the table that resolves rax to the procedures when you call int 0x80? For syscall, sys_write is rax=1

Answer 1

You question covers a number of topics, I’ll try to address them all.

1. I’m not sure there’s a single, canonical term for the way in which system calls are invoked, even less so for a specific way in which system calls are invoked (interrupt 0x80 as opposed to SYSENTER or SYSCALL). On x86-64, the documented system call interface, using SYSCALL, is described in the System V x86-64 ABI, but that’s informative only, not normative. Likewise, while most people would understand what you were talking about if you referred to it as the “i386 Linux kernel ABI” (replacing “i386” with whatever architecture you’re talking about), that could be confusing too since “kernel ABI” has another meaning (in the context of kernel modules), and again that’s not limited to interrupt 0x80.

   In practice most people shouldn’t be concerned about the specifics down to this level of detail anyway, especially since they can evolve: interrupt 0x80, SYSCALL etc. as you mention, but also the vDSO which introduces its own subtleties and is the preferred entry point for all system calls on x86 nowadays... Of course this doesn’t mean that there can’t be a term to refer to a specific calling convention, but I’m not sure it would be all that useful.

2. Windows also supports using an interrupt for its system call interface, 0x2E, but its “calling convention” is quite different: arguments are pushed on the stack, the requested system call is given by EAX, and EBX points to the arguments on the stack.

3. Current x86 kernels define the system call interface in arch/x86/entry: entry_32.S contains the i386 interface, entry_64.S the x86-32 and x86-64 interface, entry_64_compat.S the 32-bit x86-64 interface (for backward compatibility), syscalls/syscall_32.tbl the i386 system call table, syscalls/syscall_64.tbl the x86-32 and x86-64 system call table.

   The comments in those files document the interface, in particular how arguments are passed: for 32-bit calls, EAX contains the system call number, and its parameters are placed in EBX, ECX, EDX, ESI, EDI, and EBP (the parameter itself for SYSENTER, a pointer to the user stack containing the parameter for interrupt 0x80); for 64-bit calls, RAX contains the system call number, and its parameters are placed in RDI, RSI, RDX, R10, R8, and R9 (see also Why did the system call registers and order change from Intel 32bit to 64bit?). There’s a nice summary with diagrams in calling.h.

As a side note, historical comparisons often refer to the MS-DOS call interface, which primarily used interrupt 0x21; it also included the multiplex interrupt, 0x2F, which provided an extensible mechanism for adding system services (typically involving TSRs; device drivers mostly used a different interface).