I want to write my own custom file system in Linux. Can I share it over standard NFS without making any changes to the local NFS server? If so what VFS APIs are required to be implemented by my file system? Any other guidelines / gotchas for such an approach?
Yes. Once linux recognizes it as a filesystem and mounts it, it can be shared over NFS like any other filesystem, whether it's a hard drive, CD, USB stick, or even another NFS filesystem shared from somewhere else.
That's what abstraction layers are for.
That are two separate tasks:
The second one is really (mostly) independent of the first one (if the kernel can mount it, it can be exported). The hard work is in the first point. And as lots of experiences with remote file systems have shown, full POSIX compliance is possible only at the price of totally unacceptable performance. Look around NFS (various versions), Remote File System, Andrew File System, and there sure are others I forget right now, for the tradeoffs considered (and mostly discarded, it if for a reason that NFS is still the remote filesystem for Unix). Also look at CIFS, the remote filesystem from the Windows world.
The first one is a major undertaking. Look at the massive amount of work poured into BTRFS by Oracle, and it is still few years away from non-experimental status. Consider the failed ReiserFS and others. The current ext4 filesystem is a relatively straightforward development of the ext filesystem, one of the first native filesystems for Linux.
I'd suggest you attach yourself to work within one of the current filesystem development groups to learn the ropes before starting on such a project.