Forget about real mode, that's just a detail of the x86 architecture and it's just there for compatibility with 1980's processors.
The processor indeed has a flag that indicates the current privilege level. The details of that flag vary between processor types but to keep things simple just think of it as two settings: user and kernel.
There are processor instructions that change the privilege level. The critical thing for security is that going to a higher privilege level also jumps to a predefined address. The address may be in a register that only kernel code is allowed to modify, or in memory that the kernel configures (via the MMU) to be inaccessible in user mode. The code at this address is careful to verify requests made by user code, it won't accept arbitrary requests from user code.
There are typically several ways of going to a higher privilege, at least three types: system calls (when user code explicitly switches to kernel mode), interrupts (when a peripheral signals the processor that it should do something), and traps (when the processor attempts to execute invalid cide, e.g. access to unmapped memory or an unknown instruction).
On a system call, the user code doesn't specify where to jump, it just issues the system call instruction and the processor determines where to jump. To determine which system call is invoked, kernel code looks at the content of registers: typically there's a register that contains a system call number, and the system call dispatcher in the kernel looks up that number in a table.
See also http://unix.stackexchange.com/questions/95876/user-space-to-kernel-space-transitionUser space to kernel space transition and http://unix.stackexchange.com/questions/211951/how-does-the-kernel-prevent-a-malicious-program-from-reading-all-of-physical-ramHow does the kernel prevent a malicious program from reading all of physical RAM?