2

I have a monster server I'm provisioning at the minute.

There are 15 x 8TB HDD's connected to a SATA interface card that I'm using to create a ZFS volume. The drives are all detected and I've got the ZFS libraries installed and they're ready etc.

I created my ZFS volume using zpool. I opted for RAIDZ2, as I want double parity for the extra fault tolerance.

zpool create -f diskpool1 raidz2 sdc sdd sde sdf sdg sdh sdi sdj sdk sdl sdm sdn sdo sdp sdq

and if I check the status, that's all good:

[root@BACKUPNAS-I ~]# zpool status
pool: diskpool1
state: ONLINE
scan: none requested
config:

    NAME        STATE     READ WRITE CKSUM
    diskpool1   ONLINE       0     0     0
      raidz2-0  ONLINE       0     0     0
        sdc     ONLINE       0     0     0
        sdd     ONLINE       0     0     0
        sde     ONLINE       0     0     0
        sdf     ONLINE       0     0     0
        sdg     ONLINE       0     0     0
        sdh     ONLINE       0     0     0
        sdi     ONLINE       0     0     0
        sdj     ONLINE       0     0     0
        sdk     ONLINE       0     0     0
        sdl     ONLINE       0     0     0
        sdm     ONLINE       0     0     0
        sdn     ONLINE       0     0     0
        sdo     ONLINE       0     0     0
        sdp     ONLINE       0     0     0
        sdq     ONLINE       0     0     0

errors: No known data errors

But if I check the disk space, I'm only showing 87Tb :(

[root@BACKUPNAS-I ~]# df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   50G  3.0G   47G   6% /
devtmpfs                 7.7G     0  7.7G   0% /dev
/dev/mapper/centos-home  154G   54M  154G   1% /home
/dev/md126p1             497M  188M  310M  38% /boot
diskpool1                 87T  256K   87T   1% /diskpool1

I know the raid levels vary from the RAID standards when using ZFS, due to its origins, however I anticipated that I'd have around 104TB usable with a RAID6-like configuration, giving me a fault tolerance of 2 disks in the pool.

Am I not doing something correctly, or is it simply the case that using what is essentially 'software RAID' with ZFS uses takes up a lot of space (2 or so of my 8TB HDD's worth!)

Improve this question
6
  • I'd suggest you to update your title. The current one is not at all expressive, as it doesn't tell what your problem is.
    – uuu
    Commented Mar 23, 2016 at 13:17
  • Are you using ZFS on the disks or are the disks partitioned and you run ZFS on the partitions? Unrelated side note: 15*8TB in a RAIDZ2 will take ages to resilver. I would recommend splitting that up into two vdevs (maybe add one or three additional disks).
    – Marco
    Commented Mar 23, 2016 at 14:43
  • Are these 8 TiB drives (8*2^40 or 8*1024^4 or 8,796,093,022,208 bytes) or 8 TB (8*10^12 or 8*1000^4 or 8,000,000,000,000 bytes) drives? most likely the latter because marketers at hard disk companies long ago decided to specify drives in powers of 10 or 1000 rather than powers of 2 or 1024 because 8TB looks bigger than 7.27TiB. Try df -H rather than df -h. read man df to discover that -h is powers-of-1024 while -H is powers-of-1000. see en.wikipedia.org/wiki/Tebibyte
    – cas
    Commented Mar 24, 2016 at 2:06
  • BTW, this is worth a read for anyone thinking about using ZFS jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-not-raidz
    – cas
    Commented Mar 24, 2016 at 2:27
  • @cas, if it were just a metric-vs-imperial-terabyte issue, df would be showing 94.5 TB, not 87TB.
    – Mark
    Commented Apr 13, 2016 at 23:42

1 Answer 1

Reset to default
2

Looks like there's two problems.

First hard drives are in TB (1000^4) while df reports in TiB (1024^4). Calculating your expected 104TB to TiB gets you to 94.5TiB. Still off.

But you said you had two drives, so (15*8)-(2*8)=104TB. What if you had a third drive in for redundancy or not in use? Then you get (15*8)-(3*8)=96TB. Convert to TiB and you get 87.3TiB which is what df is reporting to you.

Improve this answer
2
  • yes, this vdev looks like it's raidz-3 rather than raidz-2. try zpool history | grep 'zpool create' for the exact command line used to create it.
    – cas
    Commented Mar 24, 2016 at 2:36
  • So indeed the difference between TB and TiB makes up for a decent chunk of my missing space. The rest, I've found, is due to 'padding' with ZFS and there isn't a whole lot that can be done about it.
    – John
    Commented Mar 30, 2016 at 9:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .