2

I noticed plenty of times already that when host OS is under heavy i/o load all VMs start to incredibly struggle with i/o up to point where they fail to even boot with systemd giving CPU stuck messages and other timeouts. It works this way both under VMWare Workstation and VirtualBox on Linux host. With any guest. Basically if host is performing for example filesystem check or some checksum computations guest OSes are half-functional due to not working i/o almost at all.

It's worth to mention that host processes don't have problem sharing i/o during such load and when second checksumming is run i/o just gets split more or less evenly across 2 processes. Why is that? What makes VM so "special"? Can it be helped?

2
  • 1
    Are your virtual disk images files on a filesystem, or raw LVM partitions?
    – thrig
    Feb 27, 2017 at 18:42
  • 1
    I'm using vmdk pointing to raw partition on hdd (VirtualBox can produce such vmdk files. It can be imported to VMW later). On vbox I'm using dynamic disk image on ssd. But behavior under load is pretty similar.
    – Lapsio
    Feb 27, 2017 at 19:02

1 Answer 1

3

Consumer grade controllers/disks have not the capacity to perform multiple heavy I/O operations efficiently ; however as generally as fastest is the technology the process will go better (in theory).

Obviously if you running CPU/ I/O intensive operations like software RAID (as the OP said it is doing), it will degrade the performance of the whole setup, both when sharing controllers, and also using up CPU resources. I would advise investing in a hardware RAID controller at least.

Server grade hardware will generally deal better with situations of parallel heavy access of several VMs/several users.

However, going for server grade hardware is not the whole story, and there are optimisation strategies that are useful both when dealing with consumer or server grade hardware.

What can be done in VMs that help a lot into I/O taxing less the hypervisor is paravirtualizing services.

Paravirtualization means adding special drivers, that will talk directly with the virtualisation services/kernel for bulk data transfer (aka PVSCSI in VMWare parlance), and as such the actual media storage devices/NICs do not have to be emulated.

For all the vmware solutions, either Enterprise or Workstation, you have got for Linux and FreeBSD, the open-vm-tools package.

Under Debian, you install it using:

apt-get install open-vm-tools

For Debian Stretch, it does not involve compiling anything anymore. For Jessie, I do recommend installing open-vm-tools from the backports, as the backports install open-vm-tools v10.

After installing the open-vm-tools, you have to shut down the VM, and change the disk controller for type ParaVirtual, and the network controller for vmxnet3.

Have a look at Configuring disks to use VMware Paravirtual SCSI (PVSCSI) adapters (1010398)

The vmtools also allow the VMs to do memory ballooning and as such, they won't need to use up RAM they are not using.

Virtual memory ballooning is a computer memory reclamation technique used by a hypervisor to allow the physical host system to retrieve unused memory from certain guest virtual machines (VMs) and share it with others. Memory ballooning allows the total amount of RAM required by guest VMs to exceed the amount of physical RAM available on the host.

Support for emulation technologies at the processor level like VT and VT-d also help the process being smoother. Intel Virtualization Technology for Directed I/O (VT-d)

Needless to say, optimizations at the OS level that diminish I/O also help; for instance, logging to remote logging systems, and not the local VM.

Or aligning partitions.

Partition alignment in VMware vSphere 5, a DeepDrive, Part-1

Partition alignment in VMware vSphere 5, a DeepDrive, Part-2

Beware also of other I/O optimisations, like taking out database storage space out of the /var partition due to the logging daemon flushing log files to maintain log integrity in case of sudden reboots.

It also helps following the philosophy of Unix of using the minimum required services. Empirically from my use, smaller VMs will use less I/O in house keeping/paging. Obviously if you use up more memory than you have, you might have problems with I/O (aka trashing).

You can also fine-tune the I/O priority of a particular VM in the hypervisor i.e. giving it higher or less priority. I know it can be done in VCenter/VMWare ESX, probably not in VMWare Workstation as it is a level 2 hypervisor, and as such it is the host OS that deals with managing the I/O operations and slice quotas (more on that later on).

It also goes without saying that when using level 2 hypervisors that many of the optimisations we talk here should also be applied to the host OS when possible.

VMware hypervisor technology also seems to deal better with high-load I/O in multiple VMs than the alternatives.

However, if you are concerned about performance, at least in the VMWare realm, at least for production systems, I would advise going for their Type-1/bare metal hypervisor (ESX or ESXi) instead of using VMware workstation.

From Hypervisor:

Type-1, native or bare-metal hypervisors These hypervisors run directly on the host's hardware to control the hardware and to manage guest operating systems. For this reason, they are sometimes called bare metal hypervisors. The first hypervisors, which IBM developed in the 1960s, were native hypervisors.[4] These included the test software SIMMON and the CP/CMS operating system (the predecessor of IBM's z/VM). Modern equivalents include Xen, Oracle VM Server for SPARC, Oracle VM Server for x86, the Citrix XenServer, Microsoft Hyper-V and VMware ESX/ESXi. Type-2 or hosted hypervisors These hypervisors run on a conventional operating system (OS) just as other computer programs do. A guest operating system runs as a process on the host.

Type-2 hypervisors abstract guest operating systems from the host operating system. VMware Workstation, VMware Player, VirtualBox, Parallels Desktop for Mac and QEMU are examples of type-2 hypervisors.

Finally, there is also the option of going for native hypervisors or containers technologies that not add a layer of emulation to mass storage access, like Xen for Linux hosts in PV mode , docker or FreeBSD jails. This alternative also has their advantages and disadvantages, that are out of the scope of this answer.

8
  • 1
    It's not really like I'm concerned about performance it just makes me feel quite uncomfortable when VM fails to boot with blk i/o errors during VM boot when host is reading some files ._.
    – Lapsio
    Feb 27, 2017 at 13:37
  • I improved the answer. Indeed if things reach that level, you should consider having a test rig. I have a server with SSD disks, freeBSD+ZFS+bhyve just for my own personal use at work. Feb 27, 2017 at 18:53
  • The most hilarious thing is that it works like on/off switch. When I send STOP signal to heavy i/o process VMs work, when CONT they stop working. So at this point I'm just pausing i/o for VM boot and then resuming it but it's quite well... Inconveniet? at least...
    – Lapsio
    Feb 27, 2017 at 19:06
  • 1
    Are they on the same disk controller? What is the load of CPU on the host? Feb 27, 2017 at 19:12
  • 1
    IT makes sense. I have added things to the answer again. It would have had helped you including that data on the question. Feb 27, 2017 at 19:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.