0

In Debian bullseye, I am starting virtual machines using QEMU / KVM by command line (i.e., without virsh or other helpers / wrappers). One of those VMs starts from a block device that is declared as follows:

-blockdev driver=file,node-name=q1,filename=/dev/loop0 \

Just today, I accidentally noticed that QEMU gave the following warning when starting that VM:

Opening a block device as a file using the 'file' driver is deprecated

A little bit of research showed that this warning is known, and that there are solutions to it, for example @Stephen Kitt's accepted answer here which proposes the following blockdev declaration:

-blockdev node-name=q1,driver=raw,file.driver=host_device,file.filename=/dev/loop0 \

This solution undoubtedly works, but I couldn't find any documentation for file.driver=host_device. Therefore, I tested some other options and came to the following solution that also seems to work:

-blockdev driver=host_device,node-name=q1,filename=/dev/loop0 \

Could anybody please shortly explain what the difference between these two declarations is? Notably, is one of them expected to outperform the other in terms of latency or throughput?

As a bonus question, does anybody know where the documentation for host_device is? In the other question / answer linked above, there is a link to the commit that probably implemented that driver. However, I could not find any documentation behind that link either.

1 Answer 1

1

According to QEMU changelog, the deprecation of the file driver when accessing host block devices and the introduction of the host_device driver happened with QEMU release 3.0.

The PDF presentation from Stephen Kitt's answer you referred to (see pages 20...24) indicates that the most exhaustive way to define storage for QEMU would mean first setting up the data location (file or host_device) with a node_name=, then using another --blockdev option with a file=<previously_defined_node_name> to add another layer on top of the first one to specify the data format (e.g. raw or qcow2 for example). This gives excellent control to the user, but is overkill for most basic cases.

In the QEMU source code repository, the qemu-options.hx file seems to have the best description of the -blockdev options I've seen so far.

There is one paragraph that might provide some enlightenment to the file.<something>=<something> mystery:

Options that expect a reference to another node (e.g. file) can be given in two ways. Either you specify the node name of an already existing node (file=node-name), or you define a new node inline, adding options for the referenced node after a dot (file.filename=path,file.aio=native).

So the file.<something>= syntax is essentially a shorthand way to specify another -blockdev declaration, while also avoiding cluttering up the node_name namespace with "intermediate node" names.

So, Stephen Kitt's blockdev declaration:

-blockdev node-name=q1,driver=raw,file.driver=host_device,file.filename=/dev/loop0

would seem to be equivalent to an expanded form:

-blockdev node_name=<hidden_node>,driver=host_device,filename=/dev/loop0 \
-blockdev node_name=q1,driver=raw,file=<hidden_node>

The difference between that and your declaration

-blockdev driver=host_device,node-name=q1,filename=/dev/loop0 

is a trickier matter and needs a bit of diving into QEMU source code to clarify.

QEMU's -blockdev drivers file, host_device and host_cdrom have multiple versions for different host architectures. For Linux, the applicable one is located in block/file-posix.c. Search for instances of string BlockDriver bdrv_ and you'll find each one. (You'll also find that there is an entirely separate definition of host_cdrom for FreeBSD for some reason.)

Each BlockDriver seems to be defined by a structure of (mostly) function pointers. These pointers may point to a driver-specific function, or refer to a common implementation shared with another driver.

The file driver is defined like this:

BlockDriver bdrv_file = {
    .format_name = "file",
    .protocol_name = "file",
    .instance_size = sizeof(BDRVRawState),
    .bdrv_needs_filename = true,
    .bdrv_probe = NULL, /* no probe for protocols */
    .bdrv_parse_filename = raw_parse_filename,
    .bdrv_file_open = raw_open,
    .bdrv_reopen_prepare = raw_reopen_prepare,
    .bdrv_reopen_commit = raw_reopen_commit,
    .bdrv_reopen_abort = raw_reopen_abort,
    .bdrv_close = raw_close,
    .bdrv_co_create = raw_co_create,
    .bdrv_co_create_opts = raw_co_create_opts,
    .bdrv_has_zero_init = bdrv_has_zero_init_1,
    .bdrv_co_block_status = raw_co_block_status,
    .bdrv_co_invalidate_cache = raw_co_invalidate_cache,
    .bdrv_co_pwrite_zeroes = raw_co_pwrite_zeroes,
    .bdrv_co_delete_file = raw_co_delete_file,

    .bdrv_co_preadv         = raw_co_preadv,
    .bdrv_co_pwritev        = raw_co_pwritev,
    .bdrv_co_flush_to_disk  = raw_co_flush_to_disk,
    .bdrv_co_pdiscard       = raw_co_pdiscard,
    .bdrv_co_copy_range_from = raw_co_copy_range_from,
    .bdrv_co_copy_range_to  = raw_co_copy_range_to,
    .bdrv_refresh_limits = raw_refresh_limits,
    .bdrv_attach_aio_context = raw_aio_attach_aio_context,

    .bdrv_co_truncate                   = raw_co_truncate,
    .bdrv_co_getlength                  = raw_co_getlength,
    .bdrv_co_get_info                   = raw_co_get_info,
    .bdrv_get_specific_info             = raw_get_specific_info,
    .bdrv_co_get_allocated_file_size    = raw_co_get_allocated_file_size,
    .bdrv_get_specific_stats = raw_get_specific_stats,
    .bdrv_check_perm = raw_check_perm,
    .bdrv_set_perm   = raw_set_perm,
    .bdrv_abort_perm_update = raw_abort_perm_update,
    .create_opts = &raw_create_opts,
    .mutable_opts = mutable_opts,
};

So it is essentially a clone of the raw driver, referring back to it on almost all functions.

The host_device definition is just a bit more complex:

static BlockDriver bdrv_host_device = {
    .format_name        = "host_device",
    .protocol_name        = "host_device",
    .instance_size      = sizeof(BDRVRawState),
    .bdrv_needs_filename = true,
    .bdrv_probe_device  = hdev_probe_device,
    .bdrv_parse_filename = hdev_parse_filename,
    .bdrv_file_open     = hdev_open,
    .bdrv_close         = raw_close,
    .bdrv_reopen_prepare = raw_reopen_prepare,
    .bdrv_reopen_commit  = raw_reopen_commit,
    .bdrv_reopen_abort   = raw_reopen_abort,
    .bdrv_co_create_opts = bdrv_co_create_opts_simple,
    .create_opts         = &bdrv_create_opts_simple,
    .mutable_opts        = mutable_opts,
    .bdrv_co_invalidate_cache = raw_co_invalidate_cache,
    .bdrv_co_pwrite_zeroes = hdev_co_pwrite_zeroes,

    .bdrv_co_preadv         = raw_co_preadv,
    .bdrv_co_pwritev        = raw_co_pwritev,
    .bdrv_co_flush_to_disk  = raw_co_flush_to_disk,
    .bdrv_co_pdiscard       = hdev_co_pdiscard,
    .bdrv_co_copy_range_from = raw_co_copy_range_from,
    .bdrv_co_copy_range_to  = raw_co_copy_range_to,
    .bdrv_refresh_limits = raw_refresh_limits,
    .bdrv_attach_aio_context = raw_aio_attach_aio_context,

    .bdrv_co_truncate                   = raw_co_truncate,
    .bdrv_co_getlength                  = raw_co_getlength,
    .bdrv_co_get_info                   = raw_co_get_info,
    .bdrv_get_specific_info             = raw_get_specific_info,
    .bdrv_co_get_allocated_file_size    = raw_co_get_allocated_file_size,
    .bdrv_get_specific_stats = hdev_get_specific_stats,
    .bdrv_check_perm = raw_check_perm,
    .bdrv_set_perm   = raw_set_perm,
    .bdrv_abort_perm_update = raw_abort_perm_update,
    .bdrv_probe_blocksizes = hdev_probe_blocksizes,
    .bdrv_probe_geometry = hdev_probe_geometry,

    /* generic scsi device */
#ifdef __linux__
    .bdrv_co_ioctl          = hdev_co_ioctl,
#endif

    /* zoned device */
#if defined(CONFIG_BLKZONED)
    /* zone management operations */
    .bdrv_co_zone_report = raw_co_zone_report,
    .bdrv_co_zone_mgmt = raw_co_zone_mgmt,
    .bdrv_co_zone_append = raw_co_zone_append,
#endif
};

It also refers back to the raw driver for big parts of its functionality, but has its own dedicated functions for things like:

  • probing for devices
  • parsing the file name (presumably to check for a valid host device)
  • opening the device (if you dig a little deeper, you'll find that accessing block devices as a user on MacOS requires quite a bit of extra code, so this is most likely required by OS-specific things like that)
  • quickly writing large blocks of zeroes
  • discarding data
  • checking and setting device permissions
  • getting device statistics
  • detecting the device's block size and drive geometry
  • passing through generic SCSI commands

Some of these host_device specific functions just perform an extra error check and then call the equivalent function of the raw driver. But it is probably the last ones that provide most of the value of the host_device driver: it allows the VM to explicitly see the actual block size and geometry (if applicable) of the host device. Passing through generic SCSI commands allows the VM to be granted access to things like tape library robots, high-capacity tape drives and DVD burners... and a lot of other things.

Your shorter declaration works because the host_device declaration inherently falls back to the functions of the raw driver in most things.

Note that the above applies only to POSIX-style operating systems: if you run QEMU over Windows, device access might have to be completely different from regular file access, and your shorter declaration might not work at all there. It could be that the distinction between file and host_device was created mostly to enable porting QEMU to other system architectures, where device access is very different from regular files.

You were worried about the performance, but I would expect that to be essentially identical, since all three drivers end up calling the exact same piece of code to implement the core functionality, the bdrv_co_preadv and bdrv_co_pwritev functions.

When layering drivers on top of each other, it would be very simple to detect if a given driver uses the exact same function as the driver that's going to be layered on top of it, and optimize away the duplication. My first guess would be that such an optimization would actually be very necessary, to avoid various kinds of silly behaviors. So if the configuration ends up doing the exact same thing, I would expect it to perform the same, no matter how it's declared.

3
  • Wow. Thank you very much for that in-depth explanation, accepted and +1. One last question, though: Is there any chance to find a complete list of options for the host_device driver? For example, I use the same options as with the file driver before (in my case, aio=native,cache.direct=on,discard=unmap in addition to those already mentioned), but actually I don't know whether there is more to it and whether these ones behave the same as with the file driver.
    – Binarus
    Aug 3, 2023 at 9:07
  • 1
    The qemu-options.hx file lists first the common blockdev driver options, then driver-specific options for the file, raw and qcow2 drivers, and then says: "Please refer to the QAPI documentation of the blockdev-add QMP command." for the rest. So, the qapi/block_core.json file (that includes comments used for document generation) uses the BlockdevOptionsFile block for each of file, host_device and host_cdrom drivers, suggesting that the same options apply to all of them.
    – telcoM
    Aug 3, 2023 at 11:23
  • Thank you very much, and +1. That was my impression as well (from playing around). Very good that it is now backed by theory.
    – Binarus
    Aug 4, 2023 at 5:38

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .