Try setting the niceness value of the program you are using to listen to the music to a lower number thus increasing it's priority ie. "10 is better then 12 and -20 is highest priority."
nice values are -20 to 19
Example: nice -n 10 (program you want to launch to listen to music)
(This will launch the program with a priority of 10)
You could also try turning off atime for the drives you have other than root on both the host and the VM. In the fstab set noatime as a option for each of the drives/partitions you wish to prevent accesstimes from being written. This should help speed up reads and writes.
If you have a large amount of ram on the host you could also try turning the swappiness down: sudo sysctl vm.swappiness=10
if that helps some more make it permanent edit /etc/sysctl.conf add vm.swappiness = 10
to the end of the file on it's own line.
After some experimenting of my own, I have found that if you can, changing from Analog Stereo Duplex to Analog Stereo Out helps also.