What is a kernel?
In the sense of your question, it is a single large program that runs at a special privilege level on the processor. It provides all of the core operating system facilities: multitasking, IPC, file systems, etc. It is also the process that runs the device drivers, which in turn control the computer's hardware on behalf of the kernel.
I understand it's the operating system
Actually, no. An operating system is much more than just the kernel.
Even back in the days when Maurice Bach was writing his book, the OS included shells, compilers, utilities, text editors, etc.
Over time, the term OS has come to include even more things, like the GUI subsystem.
It's a personal decision where you draw the line between the OS and normal user programs. Most people would agree that a GUI word processor is not part of the OS even if it was installed along with the OS proper. But, many would also agree that the plain text editor that came with the OS is part of the OS. Many in that camp would also agree that the markup processors that come with the OS — troff, TeX, etc. — are also considered OS facilities these days. But combine a text editor and a markup processor, and you have something indistinguishable from a word processor in some ways.
Drawing a stark line that everyone can agree on in impossible.
it is a process.
Not really, no.
A microkernel architecture is as close as you're going to get to making that statement true. Even then, the kernel is a collection of processes, one of which is special in that it is the one that can run all of the other processes. So even in that case, there is still a core — a tiny kernel — that cannot itself be said to be a normal process.
In the case of a monolithic kernel, the kernel is in a special position, and it runs all of the processes.
What actually happens in a System call?
Read the rest of Bach's book.
You will notice that this answer has many Wikipedia links, and most of the articles I link to are long and complicated, with many more links leading off. This is because you've basically asked us to distill a very complicated topic into a simple answer. There is not a simple answer, so I have tried to provide a guide to the answers, plural.
Does the user process go to sleep and the operating System Executes it on behalf of User process and returns the value to user process…?
In a classic monolithic kernel as discussed by Bach, yes.
Modern systems fuzz this simple picture, though.
First, the "operating system" doesn't execute the system call, the kernel does. I am not just being pedantic. Since the huge bulk of a modern OS is composed of assorted user-space programs, and modern OSes are multitasking, you can't say that the OS just stops and runs the system call. The OS may be doing many things at the same time, one of which is handling a single user program's system call.
But second, and this is much more important, modern OS kernels are no longer single-tasking programs that handle one system call at a time. An OS may be in the middle of many system calls at once. A single-tasking user space program that makes a system call may perceive that the world stops until the system call finishes, but the kernel may be doing many other things while that system call proceeds.
Even in the case of an old-style single-tasking kernel, you had things like driver top- and bottom-halves, which allowed the kernel to go off and handle things like disk I/O in order to provide low-level service to a relatively high-level system call like
or the user process executes in kernel Mode?
You could look at it that way, but it's only true in the same sort of way that my web browser and Stack Exchange are the same program because they are interoperating to provide a single cohesive experience.