When making a VM, why would I want to not
pass it all my cpu cores?
By the title of your question, seems like you are mixing two completely different things. "Passing" would mean dedicating it to the exclusive use by the VM. That is not what you're doing when you are setting the number of "virtual CPUs" in Virtualbox. And although the traditional wisdom has been for that number to be no more than the number of physical cores on the host, my experience has proven that this is not necessarily the case. For example, in Virtualbox, I have started a few VMs with 6 virtual CPUs each, on a host thas a 4-core-8-thread CPU. Some of these VMs are doing some heavy compiling tasks, and at the same time I am doing my usual work on the host. No negative effects whatsoever -- I feel no lag, no stutter, no slowdown on the host. As to the benefit, there is a measurable, though marginal speed improvement over 4-CPU guests -- the kernel compilation tasks finish measurably faster, though not by much. Further increasing the number of virtual CPUs in the guest has shown me to bring no further benefit. So, again, in this setup we are not "passing" CPU cores to the guest -- it is up to qemu/kvm/Virtualbox how to present virtual CPUs to the guests, and apparently Virtualbox is doing a good job of it.
(Note: In your post you also mention the word "Xen". I can not comment on Xen, but I have a suspicion that with Xen it may actually be a matter of "passing" CPU cores. Somebody else may want to comment on Xen.)