3

This is a question regarding the available space as reported by zfs list that does not meet user expectations. I have looked at similar questions, but I still don't understand the zfs list output. Let me explain why:

I have a simple setup with just 4x WD30EFRX disks. Each disk is reported by Linux to have the following capacity:

5860533168 512-byte logical blocks: (3.00 TB/2.72 TiB)

That is, each drive has a capacity of 3000592982016 bytes. According to the math, the aggregated raw space for the four drives is 3000592982016*4 = 12002371928064 bytes (10.9 TiB), and zpool list seems to agree:

NAME   SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH ALTROOT
nas   10.9T  1.15M  10.9T         -     0%     0%  1.00x ONLINE  -

So good so far. Now, I put those 4 drives into a RAIDZ1, meaning 25% of the total capacity should be used for parity. I would expect zfs list to report something close to 2.72 TiB *3 = 8.16 TiB. Instead, I'm getting only 7.65 TiB.

zfs list nas
NAME   USED  AVAIL  REFER  MOUNTPOINT
nas    558K  7.65T   140K  /nas

zfs list nas -p
NAME    USED          AVAIL   REFER  MOUNTPOINT
nas   571392  8416155617280  142848  /nas

I also expect ZFS to use up some space for metadata and its own internal data structures and whatnot, but I'm missing over half a Terabyte of data:

Available space according to zfs list -p is

(8416155617280/1024^3) GiB = 7838 GiB

Available space after parity (3 drives of 5860533168 512-byte sectors) should be close to

5860533168*512/(1024^3)*3 GiB = 8383 GiB

8383 GiB - 7838 GiB = 545 GiB

Over half a terabyte, close to 5% of the aggregated raw disk space, is unaccounted for. Where did all that space go?

Note: I'm running Debian Jessie, ZFS 0.6.5.2-2. I did check for the quota, reservation, refquota and refreservation properties, which are all set to none. I just created this pool and there is no data in it, and of course no snapshots. Here's my full zfs properties:

zfs get all nas                                                                            
NAME  PROPERTY              VALUE                  SOURCE
nas   type                  filesystem             -
nas   creation              Tue Feb  7 17:28 2017  -
nas   used                  558K                   -
nas   available             7.65T                  -
nas   referenced            140K                   -
nas   compressratio         1.00x                  -
nas   mounted               yes                    -
nas   quota                 none                   default
nas   reservation           none                   default
nas   recordsize            128K                   default
nas   mountpoint            /nas                   default
nas   sharenfs              off                    default
nas   checksum              on                     default
nas   compression           off                    default
nas   atime                 on                     default
nas   devices               on                     default
nas   exec                  on                     default
nas   setuid                on                     default
nas   readonly              off                    default
nas   zoned                 off                    default
nas   snapdir               hidden                 default
nas   aclinherit            restricted             default
nas   canmount              on                     default
nas   xattr                 on                     default
nas   copies                1                      default
nas   version               5                      -
nas   utf8only              off                    -
nas   normalization         none                   -
nas   casesensitivity       sensitive              -
nas   vscan                 off                    default
nas   nbmand                off                    default
nas   sharesmb              off                    default
nas   refquota              none                   default
nas   refreservation        none                   default
nas   primarycache          all                    default
nas   secondarycache        all                    default
nas   usedbysnapshots       0                      -
nas   usedbydataset         140K                   -
nas   usedbychildren        418K                   -
nas   usedbyrefreservation  0                      -
nas   logbias               latency                default
nas   dedup                 off                    default
nas   mlslabel              none                   default
nas   sync                  standard               default
nas   refcompressratio      1.00x                  -
nas   written               140K                   -
nas   logicalused           120K                   -
nas   logicalreferenced     40K                    -
nas   filesystem_limit      none                   default
nas   snapshot_limit        none                   default
nas   filesystem_count      none                   default
nas   snapshot_count        none                   default
nas   snapdev               hidden                 default
nas   acltype               off                    default
nas   context               none                   default
nas   fscontext             none                   default
nas   defcontext            none                   default
nas   rootcontext           none                   default
nas   relatime              on                     temporary
nas   redundant_metadata    all                    default
nas   overlay               off                    default

Thank you!

  • 1
    What is your ashift setting? Have a look at louwrentius.com/… , I have not checked your numbers correctly, but the difference is in similar size range. – user121391 Feb 8 '17 at 13:28
  • That's an interesting comment, I didn't know about ashift! My ashift is 12 even though my HDDs seem to have 512 byte sectors. I tried rebuilding the pool with -o ashift=9 and I get a little bit of space back (7.88 TiB for ashift=9). Previously it was 7.65 TiB for ashift=12, which is the default. Even for ashift=9, there is still over 300GB of unaccounted space. – Pablo Llopis Feb 8 '17 at 14:59
  • Correction: parted reports Sector size (logical/physical): 512B/4096B. Nevertheless, I'm skeptical zfs needs to allocate for so much metadata than I'm losing over 300GB. – Pablo Llopis Feb 8 '17 at 15:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.