The processes who want to make a x32 syscall will set a bit in the system call number, which will allow the kernel to tell them apart.
From the syscall(2) manpage:
 The x32 ABI uses the same instruction as the x86_64 ABI and is
used on the same processors. To differentiate between them,
the bit mask __X32_SYSCALL_BIT is bitwise-ORed into the system
call number for system calls under the x32 ABI. Both system
call tables are available though, so setting the bit is not a
x32 is not really a separate environment; an x32 program can make x64 system calls, and vice-versa; that's different from the ia32 emulation, which can also be supported side by side with x64 and x32.
That bit is checked in the kernel via the
static inline bool in_x32_syscall(void)
if (task_pt_regs(current)->orig_ax & __X32_SYSCALL_BIT)
Finding where the kernel code implementing
mmap() is checking it is left as an exercise to the reader (it's not hard). The kernel will also set the
__X32_SYSCALL_BIT itself explicitly on the saved
RAX register (the syscall number) in the case of an execve() of an x32 binary.