I teach a hands-on class on Linux, and unfortunately, by company policy I'm not allowed to reformat the class-provided laptops, so we're going by the VirtualBox guest approach.
Ignoring all performance concerns, here are some notes / problems I noticed:
1) Bridged mode and Wireless
Some wireless cards apparently have difficulty having "dual identities", which means that our routing / firewall / networking lessons go to hell. It's a known issue - most wireless drivers do not support bridging.
bridge | The Linux Foundation - It doesn't work with my Wireless card!
This means that if you're using a wireless interface, you have to do some extra work for the guest to have a "public" IP.
2) Desktop Integration
Save for a few wallbangers in design, the modern desktop environments are pretty well thought-out, and offer some conveniences that are lost when they don't have full access to the input/output, or to the device attach/disattach tool. For instance, the Vbox menu at the bottom gets really annoying if you have a window chooser or notifications there. And some machines use the Ctrl+Alt+cursor_key shortcut (switching workspaces) as to flip the display.
I mean, compare how easy it is to disattach a USB device in GNOME, vs the equivalent number of submenus / clicks in Windows, and I know which one I'd prefer any day.
3) USB "stealing"
Sometimes, windows just doesn't want to let go of a USB drive - telling virtualbox to attach it doesn't always work - most likely when windows is reading the contents for some reason or other. And then there are some USB drives that aren't straight-up storage devices, but do a mode switch-like action to make your drives accessible - those are annoying to attach to the Linux machine.
4) Stability
It's typically easier to "break" Windows than Linux, which is why you typically want to have Linux "protect" Windows instances, rather than vice versa. I already lost a couple of work days with the staff overwriting the wrong files and ending up breaking both our installation of Virtualbox and our Linux images.
5) Command Line Tools
On Linux at least, you have the option of mucking around with Vbox disk images using qemu-nbd and the network block device.
QEMU/Images - Wikibooks, open books for an open world - Mounting an image on the host
This lets you look at and modify the contents of the guest OS disk without having to boot it, for example, if you rendered it unbootable.
You could also do things like scripting backups of VDIs - or just their contents, or changing VirtualBox "profiles" via symlinks - a lot easier on bash.