2

We have a relatively large ZFS pool (23 vdevs in raidz3, ~1PB raw). The headnode has 16 cores and 192GB of RAM. We have one ZIL in the pool and no L2ARC. The pool is running solaris 11.3

The pool can work at a constant 1GB/s throughput, 20kIOPs (with peaks of >100kIOPs), however the performance degrades in a few days:

  • the IOPs drop (zpool iostat)
  • the throughput drops
  • there are loads of zio_read_intr, zio_write_issue, zio_write_issue_high in prstat -amL

Trying to look into zilstat.sh shows no activity which is very odd: I am not sure why I can see the log device in the zpool status command, attached to the pool, but I cannot see any activity in zilstat.sh.

In addition, we have an near exact replica of that system which performs flawlessly, with the ZIL behaving as it should. The only difference is that the ZIL is accessed through a different SAS connector. I would have expected a maybe lower performance when the ZIL is using the same SAS channel as the other vdevs, but not a complete absence of activity.

Improve this question
2
  • 1
    Have you looked for differences in the output from iostat? For example, is there any significant difference in the output from when the pool is working properly and when its performance is degraded? I personally would use something like iostat -sndzx 2 and capture about 10 sec worth of output from both states.
    – Andrew Henle
    May 20, 2019 at 9:23
  • @AndrewHenle: Thank you for this, I was using the zpool iostat command, iostat provides another look at the pool. Looking into this now...
    – Cyril
    May 22, 2019 at 7:25

2 Answers 2

Reset to default
2

iostat -sndzx provided some insight as to why the performance was low (see @Andrew comment to my original question).

One of the disks in the pool reported an average service time (asvc_t column) fluctuating between 150ms and 4,999ms and a percentage of time being busy of 60 (%b column). Other drives in the pool are at 3.3ms avsc_t and 2% busy.

In spite of the above figures produced by iostat, zpool status was still reporting the drive as online, with no read or write errors.

The reason I was not seeing activity using zilstat.sh was due to the activity being zfs send/receive only at the time. send/receive using asynchronous writes on the receive side was not using the ZIL at all. The blocks were directly written to the spinning disks.

Improve this answer
1

If you're seeing no performance in zilstat.sh, I would venture to question whether someone accidentally added the device intended to be the zil as another vdev to the other 23 vdevs in the pool.

Can you provide zpool status output to confirm the current config of the pool? If someone missed adding 'log' when creating the pool, your zil isn't doing what you expected.

Improve this answer
1
  • As described in original post, RAIDZ3 + logs (actually only 20 vdevs) zpool status rtank: ``` ... c0t5000CCA232834450d0 ONLINE 0 0 0 c0t5000CCA2328719DCd0 ONLINE 0 0 0 logs c0t5000A72030079638d0 ONLINE 0 0 0 errors: No known data errors ``` The output is too long for a reply to a comment.
    – Cyril
    May 22, 2019 at 7:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.