1

I have a ZFS pool in raid Z1 that's around 90TB or so which acts as a PLEX server. I currently have ~300GB free. On average, most files are 70-80GB each.

If I copy smaller files, ~10GB, I have no issues. The transfer is successful and INode consumption is not effected.

If I copy large files, ~70GB, the Inode consumption jumps from 1% to 100%, which shouldn't be possible in the span of a single file. If I delete the file, the consumption drops back to 1%. It's not just a single large file, any file that large causes the same issue.

I've copied the files with wget (remote FTP) or directly as NFS mount, the consumption behavior is the same.

Following this guide, it seems that since the default record size was 128KB, the inodes become consumed quickly due to the large file sizes consuming them.

Is this correct? Would 1MB record size have solved this?

Pool Status

root@pve:~# zpool status pool
  pool: pool
 state: ONLINE
  scan: scrub in progress since Thu Jul 27 18:40:26 2023
        25.0T scanned at 478M/s, 23.6T issued at 450M/s, 76.1T total
        0B repaired, 30.99% done, 1 days 10:00:10 to go
config:

        NAME                                 STATE     READ WRITE CKSUM
        pool                                 ONLINE       0     0     0
          raidz1-0                           ONLINE       0     0     0
            ata-ST6000VN001-2BB186_XXXXXXXX  ONLINE       0     0     0
            ata-ST6000VN001-2BB186_XXXXXXXX  ONLINE       0     0     0
            ata-ST6000VN001-2BB186_XXXXXXXX  ONLINE       0     0     0
          raidz1-1                           ONLINE       0     0     0
            ata-ST6000VN001-2BB186_XXXXXXXX  ONLINE       0     0     0
            ata-ST6000VN001-2BB186_XXXXXXXX  ONLINE       0     0     0
            ata-ST6000VN001-2BB186_XXXXXXXX  ONLINE       0     0     0
          raidz1-2                           ONLINE       0     0     0
            ata-ST8000VN004-2M2101_XXXXXXXX  ONLINE       0     0     0
            ata-ST8000VN004-2M2101_XXXXXXXX  ONLINE       0     0     0
            ata-ST8000VN004-2M2101_XXXXXXXX  ONLINE       0     0     0
          raidz1-3                           ONLINE       0     0     0
            ata-ST8000VN004-2M2101_XXXXXXXX  ONLINE       0     0     0
            ata-ST8000VN004-2M2101_XXXXXXXX  ONLINE       0     0     0
            ata-ST8000VN004-2M2101_XXXXXXXX  ONLINE       0     0     0

errors: No known data errors
root@pve:~# zfs get all pool
NAME  PROPERTY              VALUE                  SOURCE
pool  recordsize            128K                   default

Inode Consumption (Pre-File Copy)

The share directory is where the PLEX files are kept.

root@pve:\~# df -ih
Filesystem           Inodes IUsed IFree IUse% Mounted on
pool                    87M    17   87M    1% /pool
pool/share              87M   36K   87M    1% /pool/share
pool/main               87M    12   87M    1% /pool/main

Inode Consumption (Post-File Copy)

root@pve:~# df -ih
Filesystem           Inodes IUsed IFree IUse% Mounted on
pool                     17    17     0  100% /pool
pool/share              36K   36K     0  100% /pool/share
pool/main                12    12     0  100% /pool/main
2
  • Try it and see, you can always change it back if it doesn't help. Changing a dataset's recordsize only affects files created/modified after the change, it does not affect existing files in any way. If it does help, you should probably write a script to rewrite all existing files so that they also benefit from the increased recordsize (but remember that any existing snapshots will still consume the same amount of disk space in addition to the freshly re-written files!).
    – cas
    Commented Jul 29, 2023 at 1:38
  • BTW, it's generally not a good idea to have vdevs of different sizes in a pool, you should probably replace your 6TB drives with 8TB (it's not a disaster or even a problem if you don't, just sub-optimal).
    – cas
    Commented Jul 29, 2023 at 1:38

1 Answer 1

0

Inode usage reported by the statvfs() system call (what df uses under the hood) isn't very useful for ZFS.

Traditionally, filesystems have a fixed number of inodes, so its easy to report total & free. ZFS allocates more inodes (actually "dnodes", a similar ZFS-specific concept) as it needs them.

The numbers it reports to statvfs() for "free" are the number of dnodes ZFS could allocate from the remaining space on the pool (or dataset quota, if one is set). The total is then free+used. (source)

So to understand what's happening with "inodes free" here, probably you need to look at overall space usage on your pool. Are your copies filling the remaining space?

Generally, zpool list and zfs list will give you a better idea of where your space is going. Its not always easy; ZFS space predictions tend to be inexact because it has many strategies for using and not using space depending on how the pool is constructed, how the pool and datasets and configured and how much free space there really is.

Finally, recordsize won't make a difference to the number of dnodes used, but a larger blocksize can mean more wasted space on data that doesn't compress well (like video) (or if compression is off), and as discussed above, that changes the calculation for a guess at "free inodes".

But mostly: if everything is looking great apart from your apparent inode usage in df, then just don't worry about it - in practice, its meaningless.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .