Using FreeBSD 11.1-STABLE, I have a ZFS dataset configured with gzip-9 compression, as well as a recordsize of 8K. (This volume is for archiving of small files, not speed.)

zfs get all pool02/redactedStorage shows a compressratio of 1.4x which is worse than what I'd expect anecdotally, but there are a mix of text files and compressed files stored there so it wasn't alarming. Then I looked at some large-ish zip files stored on that dataset, and was confused.

The outputs of du -h and du -hA are not what I'd expect for compressed files.

e.g. I expect that a 40 MB file of zeroes would consume almost no disk space:

# dd if=/dev/zero of=testfile bs=4M count=10
# du -h testfile
512B    testfile
# du -hA testfile
 40M    testfile

But I'd expect that a 40 MB file of randomness would consume ~40 MB of disk space, as it's non-compressible (for all practical purposes). But I didn't expect it to consume nearly double the space:

# dd if=/dev/random of=testfile.rnd bs=4M count=10
# du -h testfile.rnd
 92M    testfile.rnd
# du -hA testfile.rnd
 40M    testfile.rnd

From researching this, it looks like it's indirect blocks consuming the extra space.

For testfile (zeroes):

Dataset pool02/redactedStorage [ZPL], ID 56, cr_txg 360697, 958G, 22881480 objects, rootbp DVA[0]=<0:9dd68959000:3000> DVA[1]=<0:14e1475b5000:3000> [L0 DMU objset] fletcher4 uncompressed LE contiguous unique double size=800L/800P birth=25863270L/25863270P fill=22881480 cksum=13497492df:14cc540c2b5f:e089aa02d6109:73afb0d244bcb42

    Object  lvl   iblk   dblk  dsize  lsize   %full  type
  22910497    3   128K     8K      0  40.0M    0.00  ZFS plain file
                                        168   bonus  System attributes
        dnode maxblkid: 5119
        path    /testfile
        uid     0
        gid     1004
        atime   Wed Apr 19 00:08:20 2023
        mtime   Wed Apr 19 00:08:20 2023
        ctime   Wed Apr 19 00:08:20 2023
        crtime  Wed Apr 19 00:08:20 2023
        gen     25862395
        mode    100644
        size    41943040
        parent  17938432
        links   1
        pflags  40800000004
Indirect blocks:
    [ No Indirect blocks ]

For testfile.rnd (randomness):

Dataset pool02/redactedStorage [ZPL], ID 56, cr_txg 360697, 958G, 22881480 objects, rootbp DVA[0]=<0:9dbfec9d000:3000> DVA[1]=<0:14ffe1461000:3000> [L0 DMU objset] fletcher4 uncompressed LE contiguous unique double size=800L/800P birth=25863170L/25863170P fill=22881480 cksum=13b3f2c021:15912a82ff8a:ebef1e0641453:7abda3903292dba

    Object  lvl   iblk   dblk  dsize  lsize   %full  type
  22910499    3   128K     8K  91.9M  40.0M  100.00  ZFS plain file
                                        168   bonus  System attributes
        dnode maxblkid: 5119
        path    /testfile.rnd
        uid     0
        gid     1004
        atime   Wed Apr 19 00:16:47 2023
        mtime   Wed Apr 19 00:16:48 2023
        ctime   Wed Apr 19 00:16:48 2023
        crtime  Wed Apr 19 00:16:47 2023
        gen     25862495
        mode    100644
        size    41943040
        parent  17938432
        links   1
        pflags  40800000004
Indirect blocks:
    [ 5120 Indirect blocks redacted ]

So is it 5120 indirect blocks * 128K = 640M and then those blocks are compressed such that it results in the 51.9M of overhead?

And if so, what would be the optimal way to un-mess this up? Create a new dataset with a larger recordsize and move the contents over?

Here are my dataset params:

NAME                  PROPERTY              VALUE                      SOURCE
pool02/redactedStorage  type                  filesystem                 -
pool02/redactedStorage  creation              Mon Jan 28  1:03 2019      -
pool02/redactedStorage  used                  958G                       -
pool02/redactedStorage  available             15.1T                      -
pool02/redactedStorage  referenced            958G                       -
pool02/redactedStorage  compressratio         1.40x                      -
pool02/redactedStorage  mounted               yes                        -
pool02/redactedStorage  quota                 none                       local
pool02/redactedStorage  reservation           none                       local
pool02/redactedStorage  recordsize            8K                         local
pool02/redactedStorage  mountpoint            /mnt/pool02/redactedStorage  default
pool02/redactedStorage  sharenfs              off                        default
pool02/redactedStorage  checksum              on                         default
pool02/redactedStorage  compression           gzip-9                     local
pool02/redactedStorage  atime                 on                         default
pool02/redactedStorage  devices               on                         default
pool02/redactedStorage  exec                  on                         default
pool02/redactedStorage  setuid                on                         default
pool02/redactedStorage  readonly              off                        default
pool02/redactedStorage  jailed                off                        default
pool02/redactedStorage  snapdir               hidden                     default
pool02/redactedStorage  aclmode               passthrough                local
pool02/redactedStorage  aclinherit            passthrough                inherited from pool02
pool02/redactedStorage  canmount              on                         default
pool02/redactedStorage  xattr                 off                        temporary
pool02/redactedStorage  copies                1                          default
pool02/redactedStorage  version               5                          -
pool02/redactedStorage  utf8only              off                        -
pool02/redactedStorage  normalization         none                       -
pool02/redactedStorage  casesensitivity       sensitive                  -
pool02/redactedStorage  vscan                 off                        default
pool02/redactedStorage  nbmand                off                        default
pool02/redactedStorage  sharesmb              off                        default
pool02/redactedStorage  refquota              none                       local
pool02/redactedStorage  refreservation        none                       local
pool02/redactedStorage  primarycache          all                        default
pool02/redactedStorage  secondarycache        all                        default
pool02/redactedStorage  usedbysnapshots       0                          -
pool02/redactedStorage  usedbydataset         958G                       -
pool02/redactedStorage  usedbychildren        0                          -
pool02/redactedStorage  usedbyrefreservation  0                          -
pool02/redactedStorage  logbias               latency                    default
pool02/redactedStorage  dedup                 off                        inherited from pool02
pool02/redactedStorage  mlslabel                                         -
pool02/redactedStorage  sync                  standard                   default
pool02/redactedStorage  refcompressratio      1.40x                      -
pool02/redactedStorage  written               958G                       -
pool02/redactedStorage  logicalused           501G                       -
pool02/redactedStorage  logicalreferenced     501G                       -
pool02/redactedStorage  volmode               default                    default
pool02/redactedStorage  filesystem_limit      none                       default
pool02/redactedStorage  snapshot_limit        none                       default
pool02/redactedStorage  filesystem_count      none                       default
pool02/redactedStorage  snapshot_count        none                       default
pool02/redactedStorage  redundant_metadata    all                        default

And partial output of zdb showing the relevant pool:

(Note ashift: 12 for this pool's underlying vdev.)

    version: 5000
    name: 'pool02'
    state: 0
    txg: 25383030
    pool_guid: 1288056053628670413
    hostid: 3785389258
    hostname: 'redacted'
    vdev_children: 1
        type: 'root'
        id: 0
        guid: 1288056053628670413
        create_txg: 4
            type: 'raidz'
            id: 0
            guid: 9072182531784548301
            nparity: 2
            metaslab_array: 49
            metaslab_shift: 37
            ashift: 12
            asize: 23978959699968
            is_log: 0
            create_txg: 4
            com.delphix:vdev_zap_top: 36
                type: 'disk'
                id: 0
                guid: 17108175667375824896
                path: '/dev/gptid/e07bacd6-1224-11e9-98bd-90b11c29519f'
                whole_disk: 1
                DTL: 293
                create_txg: 4
                com.delphix:vdev_zap_leaf: 37
                type: 'disk'
                id: 1
                guid: 6726950469173540573
                path: '/dev/gptid/e443f9f2-1224-11e9-98bd-90b11c29519f'
                whole_disk: 1
                DTL: 292
                create_txg: 4
                com.delphix:vdev_zap_leaf: 38
--------==== 10 ADDITIONAL PHY DISKS REDACTED ====---------            
  • FWIW, I can't reproduce on Linux on a compressed dataset in a raidz1 zpool. I see dnode maxblkid: 319 and 40MiB usage (2MiB for the output of yes | head -c40M) Apr 19, 2023 at 8:37
  • Interesting. Thanks for experimenting! I edited this question numerous times before posting, so wound up only with the tag noting FreeBSD. Just added the OS to the question; Not sure if that's the differentiating factor?
    – s.co.tt
    Apr 19, 2023 at 15:35
  • Ah sorry, that was with the default record size of 128K. With 8K, I get 65MiB and dnode maxblkid: 5119 (and 77KiB for yes | head -c40M) Apr 19, 2023 at 17:00
  • There are likely implication wrt to stripping with records that small and ashift=2 Apr 19, 2023 at 17:03
  • That was on a raidz1 pool on 5 drives above. With one on 4 drives, I get 59M. Apr 19, 2023 at 17:18

1 Answer 1


Striping 8K records over a vdev composed of 12 disks with 4K sectors (an ashift of 12) is a terrible idea, and causes an immense amount of overhead:

From OpenZFS:

https://openzfs.github.io/openzfs-docs/Basic Concepts/RAIDZ.html

Due to these inputs, if recordsize is less or equal to sector size, then RAIDZ’s parity size will be effictively equal to mirror with same redundancy. For example, for raidz1 of 3 disks with ashift=12 and recordsize=4K we will allocate on disk:

  • one 4K block of data

  • one 4K padding block

and usable space ratio will be 50%, same as with double mirror.

Another example for ashift=12 and recordsize=128K for raidz1 of 3 disks:

  • total stripe width is 3

  • one stripe can have up to 2 data parts of 4K size because of 1 parity blocks

  • we will have 128K/2 = 64 stripes with 8K of data and 4K of parity each

so usable space ratio in this case will be 66%.

The more disks RAIDZ has, the wider the stripe, the greater the space efficiency.

That text is followed by a chart which would be illegible if screenshotted and included inline here, but it shows that for a recordsize of 1x or 2x the sector size the overhead will be 67% under RAIDZ2.

According to the chart, the solution in this case is to increase recordsize to 256K, which would have a parity+padding cost of 18% on a 12 disk RAIDZ2 vdev. (A 128K recordsize would incur 24% overhead, by comparison).

But it's not that simple. The original choice of an 8K recordsize probably would have been correct for a "classic" file system, as recordsize is the maximum block size, not a fixed block size. However, there is still a penalty for a larger recordsize with relatively small files.

Increasing recordsize will only effect data created after the change, but in this case the pool only has 6% space consumed and a current compression ratio of 1.4x. Existing data can remain in situ without causing any long term capacity issues. However, in a situation where the overhead needs to be reclaimed:

https://openzfs.github.io/openzfs-docs/Performance and Tuning/Workload Tuning.html

If you change the recordsize because your application should perform better with a different one, you will need to recreate its files. A cp followed by a mv on each file is sufficient. Alternatively, send/recv should recreate the files with the correct recordsize when a full receive is done.

From a real-world experiment on the pool in question:

# zfs set recordsize=256K pool02/redactedStorage

# dd if=/dev/zero of=testfile256.40M.zeroes bs=1M count=40
# du -h testfile256.40M.zeroes
512B    testfile256.40M.zeroes

# dd if=/dev/random of=testfile256.40M.rnd bs=1M count=40
# du -h testfile256.40M.rnd
 40M    testfile256.40M.rnd

# dd if=/dev/random of=testfile256.8K.rnd bs=8192 count=1
# du -h testfile256.8K.rnd
 37K    testfile256.8K.rnd

As you can see, a 40M file is using a logical amount of space. But an 8K file is consuming 37K of space!

So recordsize should be tuned to the contents of the dataset.

Of course, it seems that the 128K default recordsize is optimal, and I just shouldn't have touched it.

# zfs set recordsize=128K pool02/redactedStorage
# cp testfile256.40M.rnd testfile128.40M.rnd
# du -h testfile128.40M.rnd
512B    testfile128.40M.rnd
# mv testfile128.40M.rnd testfile128.40M.rnd2
# du -h testfile128.40M.rnd2
 40M    testfile128.40M.rnd2

# cp testfile256.8K.rnd testfile128.8K.rnd
# mv testfile128.8K.rnd testfile128.8K.rnd2
# du -h testfile128.8K.rnd2
 19K    testfile128.8K.rnd2

This does indeed show the 8K test file using 19K of disk space, however there is a necessary amount of metadata overhead. Looking at non-compressible existing files <=8K in size, all of them show 19K of disk usage under the original recordsize=8K as well. I further tried recordsize=64K and it did not make a difference with those sample files' sizes.

Also note that the cp followed by mv is indeed necessary to create an instance of the file under the new recordsize.

This article also includes a good description of what's going on, which I'll include for posterity:


  1. Padding, disk sector size and recordsize setting: in RAID-Z, parity information is associated with each block, not with specific stripes as is the case in RAID-5, so each data allocation must be a multiple of p+1 (parity+1) to avoid freed segments being too small to be reused. If the data allocated isn’t a multiple of p+1‘padding’ is used, and that’s why RAID-Z requires a bit more space for parity and padding than RAID-5. This is a complex issue, but in short: for avoiding poor space efficiency you must keep ZFS recordsize much bigger than disks sector size; you could use recordsize=4K or 8K with 512-byte sector disks, but if you are using 4K sectors disks then recordsize should be several times that (the default 128K would do) or you could end up losing too much space.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .