Could I get ZFS to work properly in Linux?
Are there any caveats / limitations?
ZFS is not in the official Linux kernel, and never will be unless Oracle relicenses the code under something compatible with the GPL.
This incompatibility is disputed. The main arguments in favor of ZFS being allowed on Linux systems revolve around the so-called "arm's length" rule. That rule applies in this case only if ZFS is provided as a separate module from the kernel, the two communicate only through published APIs, and both code bases can function independently of each other. The claim then is that neither code base's license taints the other because neither is a derived work of the other; they are independent, but cooperate. Nevertheless, even under this interpretation, it means the ZFS modules must still be shipped separately from the Linux kernel, which is how we see it being provided today by Ubuntu.
Quite separately from the CDDL vs GPL argument, NetApp claims they own patents on some technology used in ZFS. NetApp settled their lawsuit with Sun after the Oracle buyout, but that settlement doesn't protect any other Linux distributor. (Red Hat, Ubuntu, SuSE...)
As I see it, these are your alternatives:
Use btrfs instead, as it has similar features to ZFS but doesn't have the GPL license conflict and has been in the mainline kernel for testing since 2.6.29 (released in January 2009).
The main problem with btrfs is that it's had a long history of problems with its RAID 5/6 functionality. These problems are being worked out, but each time one of these problems surfaces, it resets the "stability clock."
Another concern is that Red Hat have indicated that the next release of Red Hat Enterprise Linux will not include btrfs.
One of the reasons Red Hat is taking that position on btrfs is that they have a plan to offer similar functionality using a different technology stack they are calling Stratis. Therefore, another option you have is to wait for Stratis to appear, with 1.0 scheduled for the first half of 2018, presumably to coincide with Red Hat Enterprise Linux 8.
Use a different OS for your file server (FreeBSD, say) and use NFS to connect it to your Linux boxes
Use ZFS on FUSE, a userspace implementation, which works neatly around the kernel licensing issue at the expense of a significant amount of performance
Integrate ZFS on Linux after installing the OS.
The license conflict makes distributing the combined system outside your organization legally questionable. I am not a lawyer, but my sense is that, patent issues aside, distributing ZFS on Linux is about as worrisome as distributing non-GPL binary drivers (such as those for certain video cards) with the system. If one of these bothers you, the other should, too.
Switch to Ubuntu, which has been shipping ZFS kernel modules with the OS since 16.04. Canonical believes that it is legally safe to distribute the ZFS kernel module with the OS itself. You would have to decide whether you trust Canonical's opinion; consider also that they may not be willing to indemnify you if a legal issue comes up.
Beware that it is not currently possible to boot from ZFS with Ubuntu without a whole lot of manual hackery.
Incidentally, btrfs is also backed by Oracle, but was started years before the Sun acquisition. I don't believe the two will ever merge, or one be deprecated in favor of the other due to the license conflict and patent issue. ZFS is too popular to go away, but there will continue to be demand for a ZFS alternative.
Several answers here mention the Behlendorf ZFS port.
Keep in mind that the Behlendorf ZFS port is currently targeted towards Lustre users with extremely large filesystems. This is what Lawrence Livermore National Labs, the US Department of Energy and other research facilities need, because they run very large filesystems (100TB - multi-Petabyte systems in the near future). Lustre runs on Linux, and is running into problems when used for filesystems above certain sizes. Some people hope to solve this problem using ZFS, which is where zfsonlinux.org comes into play.
In order for ZFS to be useful for the rest of us, the ZPL (ZFS POSIX Layer) must be ported to Linux, so that administrators can interact with the filesystem. zfsonlinux.org has a development version of the ZPL, and KQ Infotech provides another implementation of the ZPL, which is a fork of the zfsonlinux.org code.
Behlendorf wants help to improve the ZPL and to merge in any changes from KQ Infotech into the zfsonlinux.org repository. If you can do this, the community will benefit greatly, and you will be a rockstar.
Oh yes, now you can! There is ZFS on Linux Project. ZFS has been successfully ported to multiple platforms and now there is a a functional Linux ZFS kernel port.
My real experience is using Ubuntu + Native ZFS - it working very stable from daily repositories.