After playing about with a couple more VMs, I've traced the problem down to VMware adding

`svga.guestBackedPrimaryAware = "TRUE"`

to the VMX file, which indicates that VMware knows that the VM has VMware tools installed. It appears that when this flag is set, VMware doesn't expose higher resolutions via VBE, so GRUB can't use higher resolutions. Removing this line from the VMX file resolves the problem for one boot, but VMware adds it back when you shut the machine down.

This behaviour does not occur if the hardware version is 10, but does occur on 11 or 12. To fix the problem permanently, remove the line `svga.guestBackedPrimaryAware = "TRUE"` from the VMX file, and change the line `virtualHW.version = "11"` or `virtualHW.version = "12"` to `virtualHW.version = "10"`.