I'm looking at Linux 5.0-rc5 source code in arch/x86/entry/syscalls/syscall_64.tbl, find that there's no separate syscall number for x32 mmap.

So how does the kernel know we're using the x32 ABI in the userspace so not to give us a mapped address beyond 4GiB?

Or in general, how do syscalls that may return an address know we're using x32 and not to return an address beyond 4GiB?


The processes who want to make a x32 syscall will set a bit in the system call number, which will allow the kernel to tell them apart.

From the syscall(2) manpage:

[5] The x32 ABI uses the same instruction as the x86_64 ABI and  is
    used  on  the  same processors.  To differentiate between them,
    the bit mask __X32_SYSCALL_BIT is bitwise-ORed into the  system
    call  number  for  system calls under the x32 ABI.  Both system
    call tables are available though, so setting the bit is  not  a
    hard requirement.

x32 is not really a separate environment; an x32 program can make x64 system calls, and vice-versa; that's different from the ia32 emulation, which can also be supported side by side with x64 and x32.

That bit is checked in the kernel via the in_x32_syscall() function:

static inline bool in_x32_syscall(void)
#ifdef CONFIG_X86_X32_ABI
        if (task_pt_regs(current)->orig_ax & __X32_SYSCALL_BIT)
                return true;
        return false;

Finding where the kernel code implementing mmap() is checking it is left as an exercise to the reader (it's not hard). The kernel will also set the __X32_SYSCALL_BIT itself explicitly on the saved RAX register (the syscall number) in the case of an execve() of an x32 binary.

  • "Both system call tables are available though, so setting the bit is not a hard requirement" what does this mean? If I don't set __X32_SYSCALL_BIT in %rax, wouldn't the kernel return me an address I can't handle? – 炸鱼薯条德里克 Feb 7 at 1:22
  • 1
    It means that a x32 program can do a plain x64 system call without that bit set, the kernel will be OK with it, as my next paragraph says. – Uncle Billy Feb 7 at 3:01

If you look at the function signature of mmap:

void *mmap(void *addr, size_t length, int prot, int flags,
           int fd, off_t offset);

you'll notice that the memory size (length) is a size_t parameter.

Now, size_t is a platform dependent type. size_t is 32-bit on a 32-bit platform and 64-bit on a 64-bit platform.

The same goes for void *. A pointer in a 32bit architetture will be a 32bit address.

So it's not the kernel which need to know, the compiler really takes care of it.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.