For example, why can't I run Ubuntu programs on Fedora or Arch, and vice versa? And on the same note, why can't I use
apt-get in Fedora or
pacman in LFS? And what about
.rpm files? Could someone please explain for a complete Linux novice?
There are several layers of the problem, mainly:
binary compatibility - software in each distribution is compiled against libraries from that distribution. These may differ in the compiler/compiler flags used for building them and/or selected features. Hence you could hit problems where some pieces of a library from distribution A being not available in the same library in distribution B. Software from A running on B could be looking for something that just isn't there.
package management - you can use several package management systems, but it's bound for troubles, because they don't know about each other. If you use
pacmanto install X, then
rpmon the same system won't know about it. And that's just the beginning - next thing is
rpmoverwriting some file from a package installed via
pacman, thus very likely breaking the package in question and likely something more (see 1.).
There are some tools that can convert packages from one format to another, but these are often rather useless exactly because of 1.
A lot also depends on what package manager you stack on top of what. Compiling your own versions on an underlying distribution is fine, you just have to keep track of the installed files yourself. Using advanced package managers on top of LSB is likely to be more problematic, because they might remove/change files for which they think they are free to do so.
They are all the same programs; it's just a matter of how they're packaged. The biggest reason is simply because you want to have the management of the packages centralized. Each distribution's package manager is not compatible with packages from a different distribution, so to brute force install a package from a different distribution (without converting it) you would have to have multiple package managers managing packages. This causes problems with each one being unaware of which packages the other might be managing. So, you will end up with a lot of conflicts between them.
On the note of using a package manager with LFS, there really isn't much stopping you from using apt, pacman, or yum on top of LFS. I wouldn't recommend it though. It could work if you were to try right now. You could compile Arch Linux's pacman, configure it to use Arch's repositories on top of an LFS system. There is a possibility that it could work if you provided everything in the "base" package. If it works now, there's no guarantee that it will continue to work in the future. You could end up with conflicting and incompatible libraries, major changes to how the filesystem is set up (like Arch's recent move from /var/run to /run). The package maintainers are expecting the system the packages are being installed on to be a certain way, so if you can guarantee that your LFS system will meet those expectations, it should work. It would require a lot of maintenance on your part though.
Just to add some small points to peterph's answer
Here I illustrate with a minimal C example how trying to run a program against a library it wasn't compiled against can fail (or not): https://stackoverflow.com/questions/2171177/what-is-an-application-binary-interface-abi/54967743#54967743
The consequence of this is that the only way to ship a reliable cross distro binary is to ship all its library dependencies with it, which is what formats such as snap, flatpak, etc. do. The result is a much larger software package than a distro-specific build, which can rely on native .so dependencies, but it is the only alternative to having distro-specific releases.
Peter does however mention one more issue in passing: you cannot ship multiple versions of certain system components easily like you can ship .so shared libraries. The X server is an example, but perhaps the kernel itself is the most fundamental one.
These are typically runtime components that are running at all times waiting for user input to respond to (server like), and which distros normally only support one of at a time. If an incompatibly with those comes up in those, then it becomes a question of how much you need to visualize to get your software running properly. The extreme case would be "you have to run a full virtual machine like QEMU", but the more you virtualize, the slower things get, so there's a tradeoff.