No, Linux commands aren't 'baked' in. A large number are part of GNU coreutils, but each one is still a separate program within this package and can ultimately be compiled on its own.
Of course the
tree command isn't anything to do with GNU, its project page can be found here - http://mama.indstate.edu/users/ice/tree/. The source code is available to download and contains build instructions (I assume you are already familiar with
gcc). A source package will also be available for your distro and may include additions/patches not available in the upstream version (but also may be an older version). Your distro will have instructions on how to compile this. In most cases if you want to code on a package, the upstream source will be the best one to go with though.
Looking at the changelog for
tree though, I see that XML output was added in version 1.6.0 (
-X option) and HTML has always been available. There are plenty programs already available to convert between the various markup languages.
xml2yaml is a specific one that would give you
pandoc is the most comprehensive one I am aware of and has support for custom formats (although creating one is a programming task in itself).
If you still want to go ahead and try develop these features directly for
tree though, I would recommend contacting the developer and asking if they would like to incorporate them. If they are interested and you can provide patches which are of a high enough quality, it is very likely that they would eventually be incorporated into the real program. From what is in the program already, it is quite likely that he would be interested in adding
json output (on the other hand though, he may feel that the program has enough feature bloat already).
A custom format seems more ill advised to me though, as it sounds like a large undertaking for such an otherwise simple tool, unless there is a very simple way to implement it. You could like with the libraries of a larger project like pandoc for conversions, but again this is ill advised for a basic tool.