When SMP support was first added to Linux, it used a "giant lock" or "BKL" (big kernel lock), this was still present right up until a few years ago. This effectively made the kernel single threaded (except I believe for hardware interrupt servicing), so no more than one syscall could be active, which of course limited performance for many types of workload.
Over time the BKL was been replaced with fine-grained locking, some system calls can run fully concurrently, others not fully. For a simple example consider allocation of PIDs and process creation in the Linux kernel. PIDs are implemented with a bitmap, allocation and freeing of PID numbers can be done with lockless atomic operations. But, maintenance of the process table is not so simple, this is done with a "tasklist" lock to ensure integrity of the relevant data structure (
To answer the questions then:
When two applications call a same system call in parallel, are they executed in parallel?
Assuming a contemporary kernel, yes. But, depending on the syscall, they may spin, yield or otherwise defer certain operations. The concurrent invocations will not be able to update the same data structures, or access the same hardware at the same time when mutexes or locks are used.
If so, what is the CPU core where the system calls execute? Does system call run on its caller's CPU core?
It will start on the calling CPU, it may be rescheduled to another CPU depending on the syscall and what's happening elsewhere in the system, unless you have set strict affinity. If you think a little about the behaviour of the getcpu() syscall it should be clearer (ignoring the fact that it may not be a real syscall on x86).
Let's assume that we have a kernel module and two applications (A and B) are calling a same function of the kernel module through ioctl (in parallel). Do they execute in parallel?
Yes, the module is expected to use fine-grained locking and other synchronisation primitives as required.
If so, what is the CPU core that services kernel module function.
Same as above, it will start on the calling CPU.
If it is not, what would be the most efficient way to provide parallel kernel module function for multiple applications?
Depending on the module (and hardware), efficiency depends on careful and minimal use of correct locking (possibly avoiding spinlocks, reducing memory copy etc), and correct use of processor affinity. In the case you're asking about where system calls are not or can not be processed concurrently, then it's hard to give a good answer. It may be possible by delegating some parts of the driver to a multi-threaded user-space daemon (I have observed this with some crypto accelerators where the accelerator itself could only perform one operation at a time, and where some small operations were faster on CPU).
The free PDF book Linux Device Drivers (3rd edition) is invaluable for this kind of work, Chapter 5 in particular: Concurrency and Race Conditions.