0

I am trying to move from LVM to ZFS as a backing store for raw virtual machine / disk images on a new server. I am using Ubuntu 18.04 and the SSDs are all Intel D3-S4510.

The problem I am experiencing is that file access on the disk image is really slow (below 50 MB/s or ~10% of the expected value). I have removed virtualization from the equation and just mounted the drives within the host and performance is still poor.

I have tried the following experiments:

  1. Use a raw SSD.
    • Copy Raw Disk - 401 MB/s [1]
    • Copy file in mounted drive 234 MB/s [2]
  2. ZFS with a ZVOL. This was very disappointing, much slower than expected. Everything I did with the ZVOL, including the initial copy of the disk image was much slower than #3.
    • Copy Raw Disk - 76 MB/s [3]
    • Copy file in mounted drive 23 MB/s [4]
  3. ZFS with a file instead of a device. The copy raw device was really good, but the mounted file performance dropped substantially.
    • Copy Raw Disk (file) - 344 MB/s [5]
    • Copy file in mounted drive 46 MB/s [6]
  4. ZFS with a file instead of a device but exposed using blktap driver. No different from #3.
    • Copy Raw Disk (file) - 343 MB/s
    • Copy file in mounted drive 48 MB/s

The performance drop from ZFS seems much higher than I'd be expecting and is so substantial that I would not be able to use ZFS and would have to go back to LVM.

Any suggestions or advice greatly appreciated.


  • For reference, the host is limited to 4G of memory, nothing else is currently running.
  • ZFS is a single drive in a single VDEV. I know this is bad, but I wanted something comparable to a single SSD.
  • the zpool was initialised with ashift=12
  • fdisk returns
    • Sector size (logical/physical): 512 bytes / 4096 bytes
    • I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Results of iostat -xm 10

[1] sdg is the raw SSD
Device            r/s     w/s     rMB/s     wMB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sdg           2603.70    0.00    325.44      0.00     0.00     0.00   0.00   0.00    0.00    0.00   1.15   127.99     0.00   0.33  85.12

[2]
Device            r/s     w/s     rMB/s     wMB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sdg           2648.30    0.30    102.50      0.00     9.70     0.00   0.36   0.00    0.00    0.00   1.60    39.63     4.00   0.17  43.84

[3] sda is the ssd in the vdev and zd0 is the zvol
Device            r/s     w/s     rMB/s     wMB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sda           8674.70    0.00     73.96      0.00     0.00     0.00   0.00   0.00    0.00    0.00   1.49     8.73     0.00   0.10  87.08
zd0           13942.50    0.00     72.62      0.00     0.00     0.00   0.00   0.00    0.05    0.00   0.73     5.33     0.00   0.04  62.64

[4]
Device            r/s     w/s     rMB/s     wMB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sda           3088.90    0.00     28.54      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.49     9.46     0.00   0.12  37.52
zd0           4580.40    0.00     22.93      0.00     0.00     0.00   0.00   0.00    0.28    0.00   1.29     5.13     0.00   0.18  84.56

[5]
%wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sda           2667.30    8.10    333.17      0.09     1.10     0.00   0.04   0.00    0.00   42.77   3.18   127.91    11.01   0.37  99.24

[6] loop0 is the mounted file device
Device            r/s     w/s     rMB/s     wMB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
loop0          998.60    0.50     42.06      0.00     0.00     0.00   0.00   0.00    4.13    8.00   6.11    43.13     4.00   0.89  89.40
sda            811.10   11.70     99.06      0.13     0.00     0.00   0.00   0.00    0.00    0.00   0.49   125.06    11.80   0.58  48.12


  • I am sorry to say, but you do not use ZFS with the right OS. And do you see strong reason to leave LVM? – Romeo Ninov Apr 22 '19 at 17:12
  • The main reason to switch to ZFS over LVM is the better snapshot functionality and the ability to incrementally send / receive changes.I have used ZFS on Linux with an SSD L2ARC and had good performance. I didn't add an L2ARC as I was using SSDs already. – David B Apr 23 '19 at 1:06
  • 1
    For anyone coming after, I had some good recommendations: - Use fio for benchmarking - Use multiple jobs (I was able to get a 16 fold improvement) - Check optimum block size on the disk (by running fio) - zfs set recordsize=8k - Gave a big improvement, need to recopy file first - zfs set compression=lz4 - Gave a marginal improvement, but saved disk space, need to recopy file first - zfs set atime=off no noticable improvement – David B May 4 '19 at 6:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.