We have a relatively large ZFS pool (23 vdevs in raidz3, ~1PB raw). The headnode has 16 cores and 192GB of RAM. We have one ZIL in the pool and no L2ARC. The pool is running solaris 11.3
The pool can work at a constant 1GB/s throughput, 20kIOPs (with peaks of >100kIOPs), however the performance degrades in a few days:
- the IOPs drop (
zpool iostat
) - the throughput drops
- there are loads of zio_read_intr, zio_write_issue,
zio_write_issue_high in
prstat -amL
Trying to look into zilstat.sh
shows no activity which is very odd: I am not sure why I can see the log device in the zpool status command, attached to the pool, but I cannot see any activity in zilstat.sh
.
In addition, we have an near exact replica of that system which performs flawlessly, with the ZIL behaving as it should. The only difference is that the ZIL is accessed through a different SAS connector. I would have expected a maybe lower performance when the ZIL is using the same SAS channel as the other vdevs, but not a complete absence of activity.
iostat
? For example, is there any significant difference in the output from when the pool is working properly and when its performance is degraded? I personally would use something likeiostat -sndzx 2
and capture about 10 sec worth of output from both states. – Andrew Henle May 20 '19 at 9:23zpool iostat
command,iostat
provides another look at the pool. Looking into this now... – Cyril May 22 '19 at 7:25