2

I have a server setup with RaidZ2 (8 disks in array) + 2 spares (+ system disk). The disks are 6TB each, 7200 RPM SAS

The machine runs Centos7 x86_64 (not a VM), and has 128 GB of RAM and 48 cores (including hyperthreading). zfs version is zfs-0.6.5.4-1.el7.centos.x86_64 .

We use it as a user interface : user log in there and usually run some compilation or plotting jobs, then submit jobs to our computing center using condor. We have ~50 users.

Lately, the accesses to the filesystem have been very slow, despite the fact that there were no radical changes in the actual load.

running iostat (early morning, low activity), I get the following :

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          9,97    0,00    0,77   19,92    0,00   69,34

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00    38,00    1,80   11,60    10,40   225,30    35,18     0,32   23,63    3,44   26,76   6,64   8,90
sdb               4,20     0,00    0,60    0,00    19,20     0,00    64,00     0,00    1,33    1,33    0,00   1,33   0,08
sdc               0,00     0,00  112,20  121,20  2845,30 10947,50   118,19     8,83   37,59   51,97   24,27   4,28 100,00
sdg               0,00     0,00   98,60  123,40  2437,30 11149,10   122,40     7,32   32,82   45,87   22,40   4,50 100,00
sdd               0,00     0,00  116,20  120,00  2857,10 10807,20   115,70     8,88   37,42   51,33   23,95   4,23 100,00
sde               0,00     0,00  105,20  122,40  2430,10 10974,90   117,79     8,03   35,33   47,56   24,83   4,39 100,02
sdh               0,00     0,00  114,00  120,00  3269,90 10807,70   120,32     9,41   40,12   57,14   23,95   4,27 100,00 
sdi               0,00     0,00  109,40  121,60  2798,60 10978,10   119,28     8,72   37,81   55,81   21,61   4,33 100,00
sdj               0,00     0,00  112,00  121,60  2860,40 11050,50   119,10     8,31   35,88   49,27   23,55   4,28 100,00
sdf               0,00     0,00    0,00   90,40     0,00  7810,40   172,80     1,37   15,17    0,00   15,17   1,59  14,36
sdk               0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00
sdl               0,00     0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00    0,00    0,00   0,00   0,00

free -m

 total       used       free     shared    buffers     cached
 Mem:        128736      81397      47338          0          8        504
 -/+ buffers/cache:      80885      47851
 Swap:       122068       2225     119843

mpstat

10:09:07     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
10:09:07     all    8,45    0,02    1,33   12,50    0,00    0,03    0,00    0,00   77,67
10:09:07       0   28,27    0,01    2,49   12,91    0,00    0,12    0,00    0,00   56,20
10:09:07       1   24,07    0,01    1,94   17,67    0,00    0,12    0,00    0,00   56,18
10:09:07       2   24,73    0,02    1,74   22,11    0,00    0,07    0,00    0,00   51,33
10:09:07       3   21,44    0,01    1,73   25,02    0,00    0,07    0,00    0,00   51,72
10:09:07       4   21,45    0,01    1,67   26,45    0,00    0,07    0,00    0,00   50,36
10:09:07       5   21,29    0,02    1,72   26,16    0,00    0,07    0,00    0,00   50,73
10:09:07       6   19,69    0,01    1,64   27,48    0,00    0,03    0,00    0,00   51,16
10:09:07       7   19,49    0,04    1,64   27,65    0,00    0,04    0,00    0,00   51,15
10:09:07       8   15,52    0,01    1,79   37,93    0,00    0,05    0,00    0,00   44,70
10:09:07       9   20,13    0,02    1,73   25,78    0,00    0,05    0,00    0,00   52,29
10:09:07      10   16,67    0,01    2,26   15,95    0,00    0,11    0,00    0,00   65,00
10:09:07      11   16,64    0,01    1,88   22,92    0,00    0,08    0,00    0,00   58,47
10:09:07      12    4,93    0,00    1,53   15,63    0,00    0,02    0,00    0,00   77,89
10:09:07      13    4,34    0,00    1,48   15,42    0,00    0,02    0,00    0,00   78,74
10:09:07      14    4,71    0,01    1,52   15,37    0,00    0,02    0,00    0,00   78,37
10:09:07      15    4,99    0,01    1,51   15,59    0,00    0,02    0,00    0,00   77,88
10:09:07      16    4,62    0,01    1,50   15,84    0,00    0,02    0,00    0,00   78,01
10:09:07      17    4,49    0,01    1,49   15,86    0,00    0,02    0,00    0,00   78,14
10:09:07      18    4,36    0,01    1,46   15,64    0,00    0,02    0,00    0,00   78,51
10:09:07      19    4,34    0,00    1,45   15,81    0,00    0,02    0,00    0,00   78,38
10:09:07      20    4,24    0,00    1,44   15,68    0,00    0,02    0,00    0,00   78,62
10:09:07      21    4,24    0,01    1,42   15,49    0,00    0,02    0,00    0,00   78,82
10:09:07      22    4,18    0,00    1,41   15,47    0,00    0,02    0,00    0,00   78,91
10:09:07      23    3,99    0,00    1,41   15,66    0,00    0,02    0,00    0,00   78,92
10:09:07      24    5,32    0,00    0,87    7,13    0,00    0,03    0,00    0,00   86,65
10:09:07      25    7,26    0,02    1,23    8,62    0,00    0,02    0,00    0,00   82,85
10:09:07      26    6,33    0,00    1,06    6,55    0,00    0,02    0,00    0,00   86,02
10:09:07      27    6,88    0,03    1,10    8,32    0,00    0,03    0,00    0,00   83,64
10:09:07      28    6,35    0,01    1,06    7,52    0,00    0,02    0,00    0,00   85,03
10:09:07      29    6,55    0,00    1,01   11,36    0,00    0,02    0,00    0,00   81,06
10:09:07      30    6,47    0,00    0,99    8,15    0,00    0,03    0,00    0,00   84,36
10:09:07      31    6,45    0,00    1,12   21,03    0,00    0,03    0,00    0,00   71,36
10:09:07      32   10,57    0,40    1,72    9,05    0,00    0,08    0,00    0,00   78,18
10:09:07      33    5,96    0,01    0,85    7,02    0,00    0,02    0,00    0,00   86,15
10:09:07      34    3,82    0,01    0,92    7,12    0,00    0,02    0,00    0,00   88,10
10:09:07      35    5,87    0,00    0,95    8,42    0,00    0,03    0,00    0,00   84,72
10:09:07      36    1,73    0,00    0,72    1,21    0,00    0,00    0,00    0,00   96,33
10:09:07      37    4,11    0,01    1,69    1,61    0,00    0,01    0,00    0,00   92,58
10:09:07      38    2,56    0,01    1,48    1,73    0,00    0,00    0,00    0,00   94,22
10:09:07      39    2,93    0,01    1,02    1,53    0,00    0,00    0,00    0,00   94,50
10:09:07      40    1,90    0,00    0,86    1,31    0,00    0,00    0,00    0,00   95,91
10:09:07      41    1,87    0,00    0,83    1,13    0,00    0,00    0,00    0,00   96,16
10:09:07      42    1,80    0,00    0,79    1,20    0,00    0,00    0,00    0,00   96,21
10:09:07      43    1,92    0,00    0,76    1,30    0,00    0,00    0,00    0,00   96,02
10:09:07      44    1,77    0,00    0,72    1,14    0,00    0,00    0,00    0,00   96,37
10:09:07      45    1,75    0,00    0,70    1,09    0,00    0,00    0,00    0,00   96,46
10:09:07      46    1,75    0,00    0,68    1,16    0,00    0,00    0,00    0,00   96,40
10:09:07      47    2,00    0,00    0,68    1,10    0,00    0,00    0,00    0,00   96,21

vmstat

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 6 10 2278560 45860460  11968 577932    0    0   951   217    0    1  8  1 78 13  0

zpool status

  pool: storage
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Fri Mar 11 10:59:26 2016
    7,89T scanned out of 33,7T at 16,0M/s, 468h51m to go
    1009G resilvered, 23,43% done
config:

NAME             STATE     READ WRITE CKSUM
storage          DEGRADED     0     0     0
  raidz2-0       DEGRADED     0     0     0
    sdc          ONLINE       0     0     0
    sdd          ONLINE       0     0     0
    sde          ONLINE       0     0     0
    replacing-3  OFFLINE      0     0     0
      old        OFFLINE      0     0     0
      sdf        ONLINE       0     0     0  (resilvering)
    sdg          ONLINE       0     0     0
    sdh          ONLINE       0     0     0
    sdi          ONLINE       0     0     0
    sdj          ONLINE       0     0     0
spares
  sdk            AVAIL   
  sdl            AVAIL   

errors: No known data errors

zpool list

NAME      SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
storage  43,5T  33,6T  9,85T         -      -    77%  1.00x  DEGRADED  -

zpool iostat

           capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
storage     33,6T  9,85T    488    124  37,1M  5,98M
storage     33,6T  9,85T    528    203  20,6M  25,0M

arcstats

6 1 0x01 91 4368 1685567518 516801482573737
name                            type data
hits                            4    617380996
misses                          4    184919553
demand_data_hits                4    347515083
demand_data_misses              4    42374865
demand_metadata_hits            4    216908143
demand_metadata_misses          4    44228265
prefetch_data_hits              4    40380612
prefetch_data_misses            4    76191344
prefetch_metadata_hits          4    12577158
prefetch_metadata_misses        4    22125079
mru_hits                        4    110301434
mru_ghost_hits                  4    66675644
mfu_hits                        4    455326805
mfu_ghost_hits                  4    55421184
deleted                         4    80153884
mutex_miss                      4    4756376
evict_skip                      4    44213146120
evict_not_enough                4    297407150
evict_l2_cached                 4    0
evict_l2_eligible               4    11649763951616
evict_l2_ineligible             4    4762036891648
evict_l2_skip                   4    0
hash_elements                   4    2318896
hash_elements_max               4    3389458
hash_collisions                 4    14240550
hash_chains                     4    148513
hash_chain_max                  4    6
p                               4    1720197302
c                               4    18401828352
c_min                           4    17179869184
c_max                           4    34359738368
size                            4    24829542936
hdr_size                        4    805709176
data_size                       4    1145901056
metadata_size                   4    7053314560
other_size                      4    15824618144
anon_size                       4    1147083264
anon_evictable_data             4    0
anon_evictable_metadata         4    0
mru_size                        4    5877126144
mru_evictable_data              4    2097152
mru_evictable_metadata          4    0
mru_ghost_size                  4    11861538816
mru_ghost_evictable_data        4    990902272
mru_ghost_evictable_metadata    4    10870636544
mfu_size                        4    1175006208
mfu_evictable_data              4    0
mfu_evictable_metadata          4    32768
mfu_ghost_size                  4    5943696896
mfu_ghost_evictable_data        4    588803072
mfu_ghost_evictable_metadata    4    5354893824
l2_hits                         4    0
l2_misses                       4    0
l2_feeds                        4    0
l2_rw_clash                     4    0
l2_read_bytes                   4    0
l2_write_bytes                  4    0
l2_writes_sent                  4    0
l2_writes_done                  4    0
l2_writes_error                 4    0
l2_writes_lock_retry            4    0
l2_evict_lock_retry             4    0
l2_evict_reading                4    0
l2_evict_l1cached               4    0
l2_free_on_write                4    0
l2_cdata_free_on_write          4    0
l2_abort_lowmem                 4    0
l2_cksum_bad                    4    0
l2_io_error                     4    0
l2_size                         4    0
l2_asize                        4    0
l2_hdr_size                     4    0
l2_compress_successes           4    0
l2_compress_zeros               4    0
l2_compress_failures            4    0
memory_throttle_count           4    0
duplicate_buffers               4    0
duplicate_buffers_size          4    0
duplicate_reads                 4    0
memory_direct_count             4    63976
memory_indirect_count           4    149390
arc_no_grow                     4    0
arc_tempreserve                 4    0
arc_loaned_bytes                4    0
arc_prune                       4    35678
arc_meta_used                   4    23683641880
arc_meta_limit                  4    34359738368
arc_meta_max                    4    34412864080
arc_meta_min                    4    16777216
arc_need_free                   4    0
arc_sys_free                    4    2109214720

I have been trying to find out what is happening for days now without any success. At the moment I can't even run my backups properly as they take ages.

I am considering adding a SSD for L2ARC to help with our random reads (I don't think ZIL would help as all our writes are asynchronous). I could also add more RAM, although it's usually not full at all.

I used mainly lsof, iotop, iostat and top as diagnostic tools without success. Any advice ?

  • Please add cat /proc/interrupts and ps felax. Are you running irqbalance? – Rui F Ribeiro Mar 17 '16 at 9:22
  • 3
    performances are expected to degrade when resilvering. There are tunings you can do to speed it up so it doesn't take those 450 hours IIRC. What version of zfs is it? – Stéphane Chazelas Mar 17 '16 at 9:57
  • 1
    You've got a lot of w/s on every disk, while for the resilvering, it should be only on sdf (and some reads (fewer) on the other disks). Is that expected? Obviously, disk I/O is saturated on those disks (100%) which is likely the bottleneck. – Stéphane Chazelas Mar 17 '16 at 10:46
  • 3
    @Jerome This might be apropos then: Storage Pool Practices for Performance ... - Keep pool capacity below 80% for best performance – Andrew Henle Mar 17 '16 at 13:22
  • 2
    If your usage is mostly random I/O or if performance is more important to you than storage capacity, consider rebuilding or replacing your pool with mirrored pairs rather than RAID-Z (RAID-Z is not fast). This is obviously more expensive and will require more disks and hot-swap bays (requires n*2 disks rather than n-2 + spares). – cas Mar 17 '16 at 23:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.