0

I have problem for past several months which was progressively getting worse and now I'm in state that if I try to do almost any intensive IO operation on my visualization host (Proxmox ve 5.1-41) like backup or even cp/rsync, dd speed of given transfer will drop to KB/s and server will hang pretty much indefinitely giving me lot of "task hanged for more than 120s" etc. For a long time I thought that it is problem with discs, I'm running 2x SSD in ZFS Raid 1 for VM storage but recently I really started to be desperate because I'm now not able to do any backups elsewhere than SSD itself (speed when copying from one pool to same pool is OK). I then tried same speed tests that I'm doing on host inside KVM/LXC and behold speeds were without single problem, no slowdowns everything working as expected.

This finding also explained why I never found out about this problem before because I was always testing performance of VMs, never thinking that performance of host would be worse than guest.

I already posted about this problem on Proxmox forums but I'm not entirely sure that It's actually fault of their system and I would love to hear what would some of you propose as test to find out what is causing this.

I already tested all Guest OS turn off and notginh changed. Machine has plenty of free resources available in normal usage. There is enough space on disc and on RAM.

CPU is: Intel Xeon E5-2620 v4

RAM: 64 GB

DATA DISKS: 2x 1TB SSD in ZFS RAID 10

BOOT DISK: 2x satadom 32 GB in ZFS RAID 10

EDIT: Only thing that will be abnormal on graphs inside Proxmox during high IO on host is Server Load which will rocket to around 50 and than most of the time all graphs will cut out because of the load. Actual CPU load and ram usage will be quite low.

Many thanks for any idea!

EDIT 2: This is stats during data transfer (with rsync) from SSD with data sdd & sde (ZFS RAID 1) to test HDD (BTRFS RAID 1) sda & sdb but actual load is on sdf & sdg (and zd0 - SWAP) which are system SSD (ZFS RAID 1). (load can be seen from second measurement)

iostat -x -d 2
Linux 4.13.13-2-pve (klaas)     01/03/2018      _x86_64_        (16 CPU)

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
loop0             0.00     0.00    0.04    2.00     0.17   869.78   850.88     0.01    4.13    5.71    4.10   1.46   0.30
sda               0.00     0.00    0.00    0.00     0.00     0.00    40.94     0.00    2.98    2.98    0.00   1.96   0.00
sdb               0.00     0.00    0.00    0.00     0.00     0.00    40.94     0.00    3.58    3.58    0.00   2.96   0.00
sdc               0.00     2.00    0.02    0.71     0.26   108.82   297.28     0.02   22.87    7.26   23.33   9.11   0.67
sdd               0.00     0.01   12.79   39.53   794.05   645.26    55.02     0.02    0.29    0.71    0.15   0.19   0.99
sde               0.00     0.00   12.80   39.00   794.16   645.26    55.58     0.02    0.30    0.72    0.17   0.20   1.04
sdf               0.00     0.00    0.88   10.16    10.27   139.85    27.22     0.13   11.66    4.42   12.28   5.96   6.57
sdg               0.00     0.00    0.89   10.39    10.32   139.85    26.63     0.14   12.53    4.38   13.24   6.41   7.23
zd0               0.00     0.00    0.04    0.24     0.16     0.94     8.00     0.02   87.75    5.03  101.71  35.04   0.97
zd16              0.00     0.00    0.33    0.46     2.82     8.71    28.95     0.00    1.17    0.28    1.80   0.11   0.01
zd32              0.00     0.00    0.03    5.96     0.77    88.80    29.88     0.00    0.19    0.31    0.18   0.02   0.01

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
loop0             0.00     0.00    0.00    0.50     0.00     2.00     8.00     0.00    0.00    0.00    0.00   0.00   0.00
sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdb               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdc               0.00     0.50    0.00    1.00     0.00     6.00    12.00     0.01    6.00    0.00    6.00   6.00   0.60
sdd               0.00     0.00   17.50   16.50    24.00   162.00    10.94     0.01    0.35    0.69    0.00   0.35   1.20
sde               0.00     0.00   16.50   16.50    18.00   162.00    10.91     0.01    0.30    0.61    0.00   0.30   1.00
sdf               0.00     0.50    0.50    2.50     0.00    22.00    14.67     2.70  754.67  792.00  747.20 333.33 100.00
sdg               0.00     0.00    2.50    3.00     8.00    30.00    13.82     0.39   73.45  128.00   28.00  35.64  19.60
zd0               0.00     0.00    0.00    1.50     0.00     6.00     8.00     3.99  728.00    0.00  728.00 666.67 100.00
zd16              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
zd32              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
loop0             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdb               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdc               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdd               0.00     0.00   20.50    6.00  1566.00   104.00   126.04     0.01    0.30    0.39    0.00   0.23   0.60
sde               0.00     0.00   20.00    6.00  1690.00   104.00   138.00     0.01    0.46    0.40    0.67   0.38   1.00
sdf               0.00     0.50   13.50   44.50    10.00   646.00    22.62     2.93   68.03   78.67   64.81  16.97  98.40
sdg               0.50     0.50   19.00   44.00    40.00   630.00    21.27     2.85   44.41   34.74   48.59  15.24  96.00
zd0               0.00     0.00    0.00   11.00     0.00    44.00     8.00     2.59  375.45    0.00  375.45  91.09 100.20
zd16              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
zd32              0.00     0.00    0.00    4.00     0.00    32.00    16.00     0.00    0.00    0.00    0.00   0.00   0.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
loop0             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdb               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdc               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdd               0.00     0.00   61.00  117.00  7028.00  3236.00   115.33     0.04    0.25    0.49    0.12   0.19   3.40
sde               0.00     0.00   40.00   84.00  4680.00  3236.00   127.68     0.07    0.55    1.20    0.24   0.40   5.00
sdf               0.00     0.50    7.00    9.50    78.00   852.00   112.73     3.64  222.18  147.71  277.05  60.61 100.00
sdg               0.00     0.00    7.00   15.50    32.00  1556.00   141.16     2.89  121.60   59.71  149.55  44.44 100.00
zd0               0.00     0.00    0.00   21.00     0.00    84.00     8.00    19.72 2074.95    0.00 2074.95  47.62 100.00
zd16              0.00     0.00    0.00    1.00     0.00     4.00     8.00     0.00    0.00    0.00    0.00   0.00   0.00
zd32              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
loop0             0.00     0.00    0.00    1.00     0.00     4.00     8.00     0.00    0.00    0.00    0.00   0.00   0.00
sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdb               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdc               0.00     0.50    0.00    1.50     0.00     8.00    10.67     0.02   13.33    0.00   13.33  13.33   2.00
sdd               0.00     0.00   10.50    4.00   832.00    50.00   121.66     0.01    0.41    0.57    0.00   0.28   0.40
sde               0.00     0.00    8.50    4.00   576.00    50.00   100.16     0.02    1.28    0.94    2.00   1.12   1.40
sdf               0.00     2.00    5.50   11.50    12.00  1534.00   181.88     2.76  160.59  110.18  184.70  58.82 100.00
sdg               0.00     1.50    6.00   13.00    48.00  1622.00   175.79     2.86  156.42  107.67  178.92  52.63 100.00
zd0               0.00     0.00    4.00   34.50    16.00   138.00     8.00    22.63  692.10  120.00  758.43  25.97 100.00
zd16              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
zd32              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
loop0             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdb               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdc               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdd               0.00     0.00    5.50   17.00     4.00   378.00    33.96     0.00    0.09    0.36    0.00   0.09   0.20
sde               0.00     0.00    7.50    6.50    42.00    98.00    20.00     0.01    0.71    0.53    0.92   0.57   0.80
sdf               0.00     1.00    7.50   11.00    28.00  1384.00   152.65     3.16  152.65  105.60  184.73  54.05 100.00
sdg               0.00     0.50    4.00    8.00    16.00   976.00   165.33     3.36  208.00  192.50  215.75  83.33 100.00
zd0               0.00     0.00    7.00   17.50    28.00    70.00     8.00    25.68  592.65  231.71  737.03  40.82 100.00
zd16              0.00     0.00    0.00    3.50     0.00    14.00     8.00     0.00    0.00    0.00    0.00   0.00   0.00
zd32              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
loop0             0.00     0.00    0.00    0.50     0.00     2.00     8.00     0.00    0.00    0.00    0.00   0.00   0.00
sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdb               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdc               0.00     0.50    0.00    1.00     0.00     6.00    12.00     0.01   10.00    0.00   10.00  10.00   1.00
sdd               0.00     0.00    3.00   66.50    14.00  1308.00    38.04     0.01    0.17    1.33    0.12   0.12   0.80
sde               0.00     0.00    2.50   57.00     0.00  1588.00    53.38     0.01    0.24    1.60    0.18   0.17   1.00
sdf               0.00     0.00    1.50    1.00     6.00   128.00   107.20     3.27 1056.80 1004.00 1136.00 400.00 100.00
sdg               0.00     0.00    0.00    0.50     0.00    64.00   256.00     3.62 2176.00    0.00 2176.00 2000.00 100.00
zd0               0.00     0.00    0.00    0.00     0.00     0.00     0.00    25.00    0.00    0.00    0.00   0.00 100.00
zd16              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
zd32              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

Other than that System SSDs have low utilization. But I don't get why it utilizes Drives that should not participate in data transfers at all.

EDIT 3: Transfer starts at second measurement, data copied from DP1 to another HHDs with BTRFS, rpool (RAID 1 ZFS SSD) is being utilized to 100 % but it does not look like it's due to actual bandwidth

 zpool iostat 2
              capacity     operations     bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
DP1          554G   334G     16     80  1.22M  1.31M
rpool       6.69G  23.1G      0     21  17.0K   286K
----------  -----  -----  -----  -----  -----  -----
DP1          554G   334G    616    195  76.7M  4.85M
rpool       6.69G  23.1G      9     38   216K  3.87M
----------  -----  -----  -----  -----  -----  -----
DP1          554G   334G  1.05K    131   133M  1.41M
rpool       6.69G  23.1G      0     29      0  3.03M
----------  -----  -----  -----  -----  -----  -----
DP1          554G   334G      0      0  4.00K      0
rpool       6.69G  23.1G      0     25      0  3.25M
----------  -----  -----  -----  -----  -----  -----
DP1          554G   334G      1      0  8.00K      0
rpool       6.69G  23.1G      0     25  2.00K  3.14M
----------  -----  -----  -----  -----  -----  -----
DP1          554G   334G      0      0  4.00K      0
rpool       6.69G  23.1G      3     26   114K  3.10M
----------  -----  -----  -----  -----  -----  -----
DP1          554G   334G      0      0  2.00K      0
rpool       6.69G  23.1G      0     20      0  2.56M
----------  -----  -----  -----  -----  -----  -----
DP1          554G   334G      0      0  2.00K      0
rpool       6.69G  23.1G      0     15  4.00K  1.94M
----------  -----  -----  -----  -----  -----  -----
DP1          554G   334G      0      0      0      0
rpool       6.69G  23.1G      0     25      0  3.19M
----------  -----  -----  -----  -----  -----  -----
DP1          554G   334G     21      0   130K      0
rpool       6.69G  23.1G      0     14      0  1.81M
----------  -----  -----  -----  -----  -----  -----
DP1          554G   334G      0      0  8.00K      0
rpool       6.69G  23.1G      0      1  2.00K   256K
----------  -----  -----  -----  -----  -----  -----
DP1          554G   334G      0      0  2.00K      0
rpool       6.69G  23.1G      0     12      0  1.62M
----------  -----  -----  -----  -----  -----  -----
DP1          554G   334G      0      0      0      0
rpool       6.69G  23.1G      1     18  8.00K  2.37M
----------  -----  -----  -----  -----  -----  -----
DP1          554G   334G      0      0      0      0
rpool       6.69G  23.1G      8     15  84.0K  2.00M

Its related of course to some IO problems, because even when I stop the transfer, host (Proxmox GUI) will freeze and not respond for like 5-15 minutes and commands like df in cli will not respond at all for same period of time. All VMs running on machine are working as expected without any slowdown.

Amount of data actually writen to System SSDs is so small that used space (21 %) and swap usage (360 MB out of 3,6 GB - swapiness set to 10) will barely change.

Also I also tried to change disc scheduler multiple times right now I'm on noop.

I noticed that when IO'm watching top there is z_wr_iss multiple times running for longer time

PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
1967 root       1 -19       0      0      0 S   0.3  0.0   5:15.03 z_wr_iss
1969 root       1 -19       0      0      0 S   0.3  0.0   5:14.76 z_wr_iss
1974 root       1 -19       0      0      0 S   0.3  0.0   5:14.56 z_wr_iss
1975 root       1 -19       0      0      0 S   0.3  0.0   5:14.71 z_wr_iss
1981 root       0 -20       0      0      0 S   0.3  0.0   4:02.77 z_wr_int_1
1984 root       0 -20       0      0      0 S   0.3  0.0   4:02.33 z_wr_int_4
1986 root       0 -20       0      0      0 S   0.3  0.0   4:02.29 z_wr_int_6

Right now I'm not able to run iotop because system will start freezing as soon as I run it, because It's still slowed down from previous tests.

OK I'm sorry It's probably caused byZFS problem as posted in @Mark answer because as I newer see it before now when I run iotop

3268 be/0 root        0.00 B/s    0.00 B/s  0.00 % 99.99 % [z_null_int]

Is definitely there.

1

It sounds as though you are having a similar problem described by various people over the last 8 or so months. In essence the version of ZFS shipped with Proxmox 5.1 is reported to have a bug which in certain circumstances results in crippling high IO. (search z_null_int high disk I/O #6171)

Two current options are to run Proxmox 4.1 (ZFS 0.6) or use an alternate file system on your proxmox 5.1 host.

  • Please check my updated question, I already checked this problem in the past but to my understanding it talks about high utilization on disc that being used, I'm experiencing utilization only on system disc regardless where actual data transfer is. – JKalousek Jan 5 '18 at 7:27
0

Regarding the 32Gb SATA Doms being used in what I understand is an independent pool comprising of 1 mirrored VDEV. How full is this pool getting during your RSync, backup or other operations and how much SWAP is the system using? If you have not already done so consider using some of ZFS tools (zpool iostat) which may provide additional information into what is happening. (https://docs.oracle.com/cd/E19253-01/819-5461/6n7ht6r01/index.html)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.