I have a monster server I'm provisioning at the minute.

There are 15 x 8TB HDD's connected to a SATA interface card that I'm using to create a ZFS volume. The drives are all detected and I've got the ZFS libraries installed and they're ready etc.

I created my ZFS volume using zpool. I opted for RAIDZ2, as I want double parity for the extra fault tolerance.

zpool create -f diskpool1 raidz2 sdc sdd sde sdf sdg sdh sdi sdj sdk sdl sdm sdn sdo sdp sdq

and if I check the status, that's all good:

[root@BACKUPNAS-I ~]# zpool status
pool: diskpool1
state: ONLINE
scan: none requested

    diskpool1   ONLINE       0     0     0
      raidz2-0  ONLINE       0     0     0
        sdc     ONLINE       0     0     0
        sdd     ONLINE       0     0     0
        sde     ONLINE       0     0     0
        sdf     ONLINE       0     0     0
        sdg     ONLINE       0     0     0
        sdh     ONLINE       0     0     0
        sdi     ONLINE       0     0     0
        sdj     ONLINE       0     0     0
        sdk     ONLINE       0     0     0
        sdl     ONLINE       0     0     0
        sdm     ONLINE       0     0     0
        sdn     ONLINE       0     0     0
        sdo     ONLINE       0     0     0
        sdp     ONLINE       0     0     0
        sdq     ONLINE       0     0     0

errors: No known data errors

But if I check the disk space, I'm only showing 87Tb :(

[root@BACKUPNAS-I ~]# df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   50G  3.0G   47G   6% /
devtmpfs                 7.7G     0  7.7G   0% /dev
/dev/mapper/centos-home  154G   54M  154G   1% /home
/dev/md126p1             497M  188M  310M  38% /boot
diskpool1                 87T  256K   87T   1% /diskpool1

I know the raid levels vary from the RAID standards when using ZFS, due to its origins, however I anticipated that I'd have around 104TB usable with a RAID6-like configuration, giving me a fault tolerance of 2 disks in the pool.

Am I not doing something correctly, or is it simply the case that using what is essentially 'software RAID' with ZFS uses takes up a lot of space (2 or so of my 8TB HDD's worth!)

  • I'd suggest you to update your title. The current one is not at all expressive, as it doesn't tell what your problem is. – toogley Mar 23 '16 at 13:17
  • Are you using ZFS on the disks or are the disks partitioned and you run ZFS on the partitions? Unrelated side note: 15*8TB in a RAIDZ2 will take ages to resilver. I would recommend splitting that up into two vdevs (maybe add one or three additional disks). – Marco Mar 23 '16 at 14:43
  • Are these 8 TiB drives (8*2^40 or 8*1024^4 or 8,796,093,022,208 bytes) or 8 TB (8*10^12 or 8*1000^4 or 8,000,000,000,000 bytes) drives? most likely the latter because marketers at hard disk companies long ago decided to specify drives in powers of 10 or 1000 rather than powers of 2 or 1024 because 8TB looks bigger than 7.27TiB. Try df -H rather than df -h. read man df to discover that -h is powers-of-1024 while -H is powers-of-1000. see en.wikipedia.org/wiki/Tebibyte – cas Mar 24 '16 at 2:06
  • BTW, this is worth a read for anyone thinking about using ZFS jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-not-raidz – cas Mar 24 '16 at 2:27
  • @cas, if it were just a metric-vs-imperial-terabyte issue, df would be showing 94.5 TB, not 87TB. – Mark Apr 13 '16 at 23:42

Looks like there's two problems.

First hard drives are in TB (1000^4) while df reports in TiB (1024^4). Calculating your expected 104TB to TiB gets you to 94.5TiB. Still off.

But you said you had two drives, so (15*8)-(2*8)=104TB. What if you had a third drive in for redundancy or not in use? Then you get (15*8)-(3*8)=96TB. Convert to TiB and you get 87.3TiB which is what df is reporting to you.

  • yes, this vdev looks like it's raidz-3 rather than raidz-2. try zpool history | grep 'zpool create' for the exact command line used to create it. – cas Mar 24 '16 at 2:36
  • So indeed the difference between TB and TiB makes up for a decent chunk of my missing space. The rest, I've found, is due to 'padding' with ZFS and there isn't a whole lot that can be done about it. – John Mar 30 '16 at 9:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.