I know there are many differences between OSX and Linux, but what makes them so totally different, that makes them fundamentally incompatible?
The whole ABI is different, not just the binary format (Mach-O versus ELF) as sepp2k mentioned.
For example, while both Linux and Darwin/XNU (the kernel of OS X) use
sc on PowerPC and
syscall on x86 for syscall entry, there's not much more in common from there on.
Darwin directs negative syscall numbers at the Mach microkernel and positive syscall numbers at the BSD monolithic kernel — see xnu/osfmk/mach/syscall_sw.h and xnu/bsd/kern/syscalls.master. Linux's syscall numbers vary by architecture — see linux/arch/powerpc/include/asm/unistd.h, linux/arch/x86/include/asm/unistd_32.h, and linux/arch/x86/include/asm/unistd_64.h — but are all nonnegative. So obviously syscall numbers, syscall arguments, and even which syscalls exist are different.
The standard C runtime libraries are different too; Darwin mostly inherits FreeBSD's libc, while Linux typically uses glibc (but there are alternatives, like eglibc and dietlibc and uclibc and Bionic).
Not to mention that the whole graphics stack is different; ignoring the whole Cocoa Objective-C libraries, GUI programs on OS X talk to WindowServer over Mach ports, while on Linux, GUI programs usually talk to the X server over UNIX domain sockets using the X11 protocol. Of course there are exceptions; you can run X on Darwin, and you can bypass X on Linux, but OS X applications definitely do not talk X.
Like Wine, if somebody put the work into
- implementing a binary loader for Mach-O
- trapping every XNU syscall and converting it to appropriate Linux syscalls
- writing replacements for OS X libraries like CoreFoundation as needed
- writing replacements for OS X services like WindowServer as needed
then running an OS X program "natively" on Linux could be possible. Years ago, Kyle Moffet did some work on the first item, creating a prototype binfmt_mach-o for Linux, but it was never completed, and I know of no other similar projects.
(In theory this is quite possible, and similar efforts have been done many times; in addition to Wine, Linux itself has support for running binaries from other UNIXes like HP-UX and Tru64, and the Glendix project aims to bring Plan 9 compatiblity to Linux.)
Somebody has put in the effort to implement a Mach-O binary loader and API translator for Linux!
shinh/maloader - GitHub takes the Wine-like approach of loading the binary and trapping/translating all the library calls in userspace. It completely ignores syscalls and all graphical-related libraries, but is enough to get many console programs working.
Darling builds upon maloader, adding libraries and other supporting runtime bits.
Why OSX applications won't run natively on linux:
First of all OSX uses a different binary format than Linux, so Linux can't execute binaries compiled for OSX (the same way it can't execute binaries compiled for Windows or BSD).
Second of all, if you're talking about GUI applications, Apple's GUI toolkit Cocoa a) is only available for OSX and b) does not run on top of X11.
Why there is no equivalent of wine for OSX applications:
A lot of work had to be done before wine was even halfway usable. Since there is not as much demand for an OSX equivalent, no one has invested the same amount of effort into such a project yet.
The most important reason why OS X apps will not run on Linux is because those OSes used different syscalls.
Some previous answers mentioned libraries but that is generally not the case - Core Foundation is largely open sourced by Apple under the name CFLite and readily portable to any platform (the Windows version of iTunes actually sits atop of a Windows port of Core Foundation, and with some compiler tweaks, you can directly make CFLite using clang on an Linux distribution) and there are also open-sourced efforts to port Objective-C environment, mainly Foundation and AppKit to Linux, most notably GNUstep, a GNU implementation of OpenStep, which dated earlier than Apple's Cocoa (started off when there still was the company NeXT Computer.)
If someone is determined, they can design a loader that will capture every Mach-O syscall and translate it to the corresponding Linux syscall, as well as dynamically link those open-source library "counterparts" to the binary with appropriate ABI translating.
And just for your information, if you can obtain the source code of the Mach-O application, you may consider port it and it may turn out to be very simple. As an example, the TextEdit app bundled with OS X 10.6 can be directly recompiled linking against GNUstep after stripping a few lines of (non-critical) CF code and thus immediately available under Linux (not to mention the TextEdit shipped with GNUstep was actually a direct recompile of the TextEdit app from NeXTSTEP, the precursor to OS X as well, even retaining its "© 1995 NeXT" label). TextEdit is under BSD license.
On 8th of December 2012 new project has been launched - Darling.