6

If I use zpool to list the available space, it tells me I have over 270 GB free, yet the actual free space available (and shown by df and zfs list) is only a mere 40 GB, almost ten times less:

$ zpool list
NAME      SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
ssdtank  7.25T  6.98T   273G        -         -    21%    96%  1.00x    ONLINE  -

$ zpool iostat -v
                                      capacity     operations     bandwidth 
pool                                alloc   free   read  write   read  write
----------------------------------  -----  -----  -----  -----  -----  -----
ssdtank                             6.98T   273G      1     15   365K   861K
  ata-Samsung_SSD_860_QVO_4TB_S123  3.49T   136G      0      7   182K   428K
  ata-Samsung_SSD_860_QVO_4TB_S456  3.49T   137G      0      8   183K   434K

$ zfs list
NAME       USED  AVAIL     REFER  MOUNTPOINT
ssdtank   6.98T  40.6G       32K  /srv/tank

What does this discrepancy mean? Why do the two utilities show differing amounts of free space? More importantly, how can I access the "extra" 200 GB if it's really there?

The pool is made of two identical disks, no RAID or other setups, just added as plain vdevs to the pool and the filesystem created on top. (There are multiple filesystems inside the root one shown, but I don't think they are relevant because they all share the same root and have the same 40.6G free space).

As requested, here is the output of zfs get all: (I also updated the figures above so they all make sense as the amount of free disk space has changed today. The old figures were 257GB/27GB and today they are 273GB/40GB, which means the amount of disk space freed up since I originally posted the question has increased both figures by the same amount - i.e. zpool seems to be reporting approx 270 GB more than everything else, but it's consistently 270 GB more than whatever the actual free space happens to be at the time).

NAME     PROPERTY              VALUE                  SOURCE
ssdtank  aclinherit            restricted             default
ssdtank  acltype               off                    default
ssdtank  atime                 off                    received
ssdtank  available             40.6G                  -
ssdtank  canmount              on                     default
ssdtank  casesensitivity       sensitive              -
ssdtank  checksum              on                     default
ssdtank  compression           off                    default
ssdtank  compressratio         1.00x                  -
ssdtank  context               none                   default
ssdtank  copies                1                      default
ssdtank  createtxg             1                      -
ssdtank  creation              Sat Oct 26 21:53 2019  -
ssdtank  dedup                 off                    default
ssdtank  defcontext            none                   default
ssdtank  devices               on                     default
ssdtank  dnodesize             legacy                 default
ssdtank  encryption            off                    default
ssdtank  exec                  on                     default
ssdtank  filesystem_count      none                   default
ssdtank  filesystem_limit      none                   default
ssdtank  fscontext             none                   default
ssdtank  guid                  12757787786185470931   -
ssdtank  keyformat             none                   default
ssdtank  keylocation           none                   default
ssdtank  logbias               latency                default
ssdtank  logicalreferenced     16K                    -
ssdtank  logicalused           6.98T                  -
ssdtank  mlslabel              none                   default
ssdtank  mounted               yes                    -
ssdtank  mountpoint            /srv/tank              local
ssdtank  nbmand                off                    default
ssdtank  normalization         none                   -
ssdtank  objsetid              54                     -
ssdtank  overlay               off                    default
ssdtank  pbkdf2iters           0                      default
ssdtank  primarycache          all                    default
ssdtank  quota                 none                   default
ssdtank  readonly              off                    default
ssdtank  recordsize            128K                   default
ssdtank  redundant_metadata    all                    default
ssdtank  refcompressratio      1.00x                  -
ssdtank  referenced            32K                    -
ssdtank  refquota              none                   default
ssdtank  refreservation        none                   default
ssdtank  relatime              off                    default
ssdtank  reservation           none                   default
ssdtank  rootcontext           none                   default
ssdtank  secondarycache        all                    default
ssdtank  setuid                on                     default
ssdtank  sharenfs              rw=@192.168.0.0/24     received
ssdtank  sharesmb              off                    default
ssdtank  snapdev               hidden                 default
ssdtank  snapdir               hidden                 default
ssdtank  snapshot_count        none                   default
ssdtank  snapshot_limit        none                   default
ssdtank  special_small_blocks  0                      default
ssdtank  sync                  standard               default
ssdtank  type                  filesystem             -
ssdtank  used                  6.98T                  -
ssdtank  usedbychildren        6.98T                  -
ssdtank  usedbydataset         32K                    -
ssdtank  usedbyrefreservation  0B                     -
ssdtank  usedbysnapshots       0B                     -
ssdtank  utf8only              off                    -
ssdtank  version               5                      -
ssdtank  volmode               default                default
ssdtank  vscan                 off                    default
ssdtank  written               0                      -
ssdtank  xattr                 on                     default
ssdtank  zoned                 off                    default
2
  • Please update the question with the output of zfs get all | sort. – binarysta Apr 23 '20 at 14:10
  • @binarysta Added as requested, hope it tells you something! – Malvineous Apr 24 '20 at 11:41
6

Internally ZFS reserves a small amount of space (slop space) to ensure some critical ZFS operations can complete even in situations with very low free space.

The amount is 3.2% of total pool capacity. zfs-0.6.5

3.2% of 7.25T = 235GB

You really only have 40.6GB free in the filesystem.

zpool reports about the raw disk capacities and the free space will be 40 + 235 = 275G

1
  • That explains it! I found that you can tune this amount of reserved space so will put the details in another answer. With an absolute minimum slop space of 128 MiB, I think 235 GB is a tad excessive! – Malvineous Apr 25 '20 at 8:17
1

To add to binarysta's answer, you can tune the amount of reserved ("slop") space with the spa_slop_shift module option.

The ZFS-on-Linux module documentation suggests that the default value of 5 can be too conservative (too much reserved space) for large arrays, so it's ok to increase it, which reserves less space. (Other OSes running ZFS have similar options.)

The default of 5 translates to 2^5 = 32, or 1/32 of the pool's capacity. Setting spa_slop_shift to 6 (the value used in older ZFS versions) translates to reserving only 1/64 of the pool's capacity.

Reserving less space can increase fragmentation and can require more memory to manage a larger number of disparate blocks, which is why the docs suggest 15 (1/32768 of pool space) being the upper practical limit for a system with 4 TB of RAM. Presumably this memory requirement would be less if you could somehow guarantee that the filesystem would not become fragmented as it fills up.

With the absolute minimum slop space being 128 MiB, I think 200 GB is a bit too much so I changed spa_slop_shift to 6 for the time being. Each increment halves the amount of reserved space, so going from 5 to 6 will, in my case, halve the reserved space from 200 GB to 100 GB. Sure enough, this immediately gave me an extra 100 GiB of free space.

To change the value temporarily at runtime on Linux, you can change the module parameter without rebooting:

echo 6 > /sys/module/zfs/parameters/spa_slop_shift

This will take effect immediately, and df will show the extra space straight away.

To make the change permanent, create or edit /etc/modprobe.d/zfs.conf and add:

options zfs spa_slop_shift=6

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.