We have a hadoop cluster; some of the machines in the cluster are VM machines and all others are physical machines.
According to documentation, it is better to set MTU=9000 on all machines, but I am not sure if the VM's can support MTU=9000.
When using Jumbo frames, all the NICs in a VLAN/netblock must support and be configured to Jumbo frames without exception.
I would stress ALL of them.
In doubt, make plans and make some extensive tests with all the hosts you have.
If the tests are successful, add another NIC to those VMs and create a special VLAN just for the hadoop VMs to transfer data between them.
I would again stress out the importance of interfaces of MTU 9000 and MTU 1500 not cohabiting in the same VLAN, and furthermore of not creating Jumbo frame VLANs connected to the Internet.
When dealing with VmWare, official VmWare documentation states you need VMXNET3 interfaces [and consequently
open-vm-tools] for dealing with MTU=9000 Jumbo-enabled interfaces.
Jumbo frames also have to be configured in distributed vmware switches and in the VMWare Kernel before configuring them on the VMs. I think recently you are able to get away using them on ESxi. In the past they were only supported by the Enterprise VMWare licenses.