1

People are experiencing slowness with the server. A write to a small table took over 5 secs which is extremely slow. I have checked for corruption of the MySQL database, D3 database and Linux filesystem and no errors are reported.

I am getting group descriptors corrupted! in dmesg which appears to be reported by the Acronis SnapAPI. I have spoken to Acronis and waiting for them to get back to me on this. Hopefully, they can point me in the right direction on the location of the corruption.

The other thing in question in dmesg is sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. I am not sure what this warning means.

Top shows a high load but no process stands out as the top consumer of resources.

top - 10:38:43 up 2 days, 15:40, 27 users,  load average: 4.91, 3.71, 3.93
Tasks: 516 total,   1 running, 515 sleeping,   0 stopped,   0 zombie
Cpu(s):  3.2%us,  0.5%sy,  0.7%ni, 93.1%id,  2.3%wa,  0.0%hi,  0.2%si,  0.0%st
Mem:  10864732k total, 10715688k used,   149044k free,   124068k buffers
Swap:  8175612k total,   355552k used,  7820060k free,  6677196k cached
   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                          
 15724 mysql     20   0 5460m 480m 4252 S 11.9  4.5  50:44.93 mysqld                                                                           
 16170 pick      20   0 1261m 166m 4376 S 10.6  1.6  20:06.69 java                                                                             
 17416 pick      24   4 5951m 679m 678m S  4.6  6.4  19:56.12 d3                                                                               
 16995 root      16  -4 6025m 797m 797m S  1.7  7.5   8:24.19 d3      

CPU stays idle most of the time

[root@m10pmds autoit]# mpstat -P ALL 1 100
Linux 2.6.32-696.el6.x86_64 (m10pmds.berwickmazda.local)        10/31/2022      _x86_64_        (6 CPU)

11:06:41 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
11:06:42 AM  all    3.35    0.00    0.67    0.17    0.00    0.17    0.00    0.00   95.64
11:06:42 AM    0    7.07    0.00    2.02    0.00    0.00    0.00    0.00    0.00   90.91
11:06:42 AM    1    3.06    0.00    1.02    0.00    0.00    0.00    0.00    0.00   95.92
11:06:42 AM    2    0.99    0.00    0.99    0.00    0.00    0.00    0.00    0.00   98.02
11:06:42 AM    3    2.00    0.00    0.00    1.00    0.00    0.00    0.00    0.00   97.00
11:06:42 AM    4    2.02    0.00    0.00    0.00    0.00    0.00    0.00    0.00   97.98
11:06:42 AM    5    4.04    0.00    1.01    0.00    0.00    0.00    0.00    0.00   94.95

11:06:42 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
11:06:43 AM  all    0.67    0.00    0.17    0.00    0.00    0.00    0.00    0.00   99.16
11:06:43 AM    0    0.99    0.00    0.00    0.00    0.00    0.00    0.00    0.00   99.01
11:06:43 AM    1    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
11:06:43 AM    2    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
11:06:43 AM    3    1.01    0.00    0.00    0.00    0.00    0.00    0.00    0.00   98.99
11:06:43 AM    4    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
11:06:43 AM    5    3.06    0.00    0.00    0.00    0.00    0.00    0.00    0.00   96.94

11:06:43 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
11:06:44 AM  all    6.23    0.00    0.67    0.00    0.00    0.34    0.00    0.00   92.76
11:06:44 AM    0    6.06    0.00    0.00    0.00    0.00    0.00    0.00    0.00   93.94
11:06:44 AM    1    8.08    0.00    1.01    0.00    0.00    1.01    0.00    0.00   89.90
11:06:44 AM    2    7.00    0.00    1.00    0.00    0.00    1.00    0.00    0.00   91.00
11:06:44 AM    3   14.14    0.00    2.02    0.00    0.00    1.01    0.00    0.00   82.83
11:06:44 AM    4    1.00    0.00    1.00    0.00    0.00    0.00    0.00    0.00   98.00
11:06:44 AM    5    0.99    0.00    0.99    0.00    0.00    0.00    0.00    0.00   98.02

Memory doesn't appear to have any problems:

[root@m10pmds autoit]# free -h
             total       used       free     shared    buffers     cached
Mem:           10G        10G       137M       5.7G       121M       6.4G
-/+ buffers/cache:       3.8G       6.6G
Swap:         7.8G       347M       7.5G

Nothing particular stands out with iotop. There are small reads and writes every second or two and it ranges in the kb size. No swap in or out.

Total DISK READ: 0.00 B/s | Total DISK WRITE: 49.05 K/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND                                                                         
  461 be/3 root        0.00 B/s    0.00 B/s  0.00 %  0.19 % [jbd2/dm-0-8]
34317 be/4 mysql       0.00 B/s    3.77 K/s  0.00 %  0.13 % mysqld --basedir=/usr --datadir=/var/lib/~ld.pid --socket=/var/lib/mysql/mysql.sock
167595 be/4 pick        0.00 B/s    3.77 K/s  0.00 %  0.00 % java -DSTART=extra/etc/start-plus.config~/lib/jetty -jar start.jar etc/desktop.xml
161564 be/4 pick        0.00 B/s    3.77 K/s  0.00 %  0.00 % java -DSTART=extra/etc/start-plus.config~/lib/jetty -jar start.jar etc/desktop.xml
    1 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % init
    2 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kthreadd]
    3 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/0]
    4 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/0]
    5 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [stopper/0]
    6 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/0]
    7 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/1]
    8 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [stopper/1]
    9 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/1]
   10 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/1]
   11 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/2]
   12 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [stopper/2]
   13 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/2]
   14 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/2]
   15 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/3]
   16 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [stopper/3]

Processes are frequently in the D state


[autoit@m10pmds ~]$ ps aux | awk '$8 ~ /D/'
root        224  0.0  0.0      0     0 ?        D    Oct28   0:00 [scsi_eh_0]
root        461  0.0  0.0      0     0 ?        D    Oct28   0:41 [jbd2/dm-0-8]
root       1089  0.0  0.0      0     0 ?        D    Oct28   0:01 [jbd2/dm-3-8]
root       1388  0.0  0.0      0     0 ?        D    Oct28   0:14 [flush-253:0]
root      16975  0.0  0.0  45224   480 ?        D    Oct28   0:43 udisks-daemon: polling /dev/sr0
pick      17425  0.0  3.1 6093852 344572 ?      DN   Oct28   4:35 d3 -n pick0 133 -phantom
pick      17432  0.8 36.1 6093852 3930268 ?     DN   Oct28  46:54 d3 -n pick0 135 -phantom
pick      17436  0.0  6.4 6093852 700592 ?      DN   Oct28   2:07 d3 -n pick0 136 -phantom
[autoit@m10pmds ~]$ ps aux | awk '$8 ~ /D/'
root        224  0.0  0.0      0     0 ?        D    Oct28   0:00 [scsi_eh_0]
root        461  0.0  0.0      0     0 ?        D    Oct28   0:41 [jbd2/dm-0-8]
root       1089  0.0  0.0      0     0 ?        D    Oct28   0:01 [jbd2/dm-3-8]
root       1093  0.0  0.0      0     0 ?        D    Oct28   0:11 [jbd2/dm-5-8]
root       1388  0.0  0.0      0     0 ?        D    Oct28   0:14 [flush-253:0]
root      16975  0.0  0.0  45224   480 ?        D    Oct28   0:43 udisks-daemon: polling /dev/sr0
pick      17425  0.0  3.1 6093852 344572 ?      DN   Oct28   4:35 d3 -n pick0 133 -phantom
pick      17432  0.8 36.1 6093852 3930268 ?     DN   Oct28  46:54 d3 -n pick0 135 -phantom
pick      17436  0.0  6.4 6093852 700592 ?      DN   Oct28   2:07 d3 -n pick0 136 -phantom

Process 17432 seems to be in the D status most frequently compared to other processes:

[root@m10pmds autoit]# ps aux | awk '$8 ~ /D/'
pick      17432  0.9 36.1 6093852 3930268 ?     DN   Oct28  50:12 d3 -n pick0 135 -phantom

wa is high quite frequently but no process stands out as using up I/O:

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  7 357964 140112 126960 6653332    0    0     0    32  922 1365  1  0 17 82  0
 0  7 357964 140112 126960 6653332    0    0     0     0  913 1499  1  0 16 83  0
 0  7 357964 139988 126960 6653332    0    0     0     0 1147 1924  1  0 16 82  0
 0  7 357964 140276 126960 6653332    0    0     0     0  838 1369  1  0 16 83  0
 0  7 357964 140268 126960 6653332    0    0     0     0 1327 2291  2  0 16 82  0
 6  1 357964 139244 126968 6653324    0    0     4   704 1446 2570  3  1 17 79  0
 0  3 357964 136452 126972 6653392    0    0   567  2812 3117 3685 13  5 64 19  0
 2  3 357964 136360 126972 6653352    0    0     0     0  806 1367  0  0 50 50  0
 0  4 357964 136196 126972 6653352    0    0     4     0 1457 2589  1  1 34 64  0
 0  5 357964 136320 126972 6653352    0    0     0     4  843 1363  2  0 21 77  0
 0  6 357964 136196 126972 6653352    0    0     0    64 1197 1941  2  0 16 82  0
 0  8 357964 136212 126976 6653352    0    0     0    28 1307 1385  9  0  9 82  0
 0  9 357964 135824 126976 6653352    0    0     0     4 1126 1441  5  1 17 77  0
 0  9 357964 135544 126976 6653356    0    0     0     0  984 1481  1  0 16 82  0
 0 11 357964 135636 126976 6653360    0    0     0     8  932 1634  1  0 17 82  0

Network isn't saturated and data transfer is minimal.

10:54:14 AM     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
10:54:15 AM        lo    238.14    238.14    180.82    180.82      0.00      0.00      0.00
10:54:15 AM      eth0     70.10     62.89     12.19     70.07      0.00      0.00      0.00

10:54:15 AM     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
10:54:16 AM        lo    318.00    318.00    303.95    303.95      0.00      0.00      0.00
10:54:16 AM      eth0     39.00     29.00      6.16     49.79      0.00      0.00      0.00

10:54:16 AM     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
10:54:17 AM        lo     69.39     69.39     78.91     78.91      0.00      0.00      0.00
10:54:17 AM      eth0     62.24     24.49     12.56     18.28      0.00      0.00      0.00

10:54:17 AM     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
10:54:18 AM        lo     85.00     85.00     33.37     33.37      0.00      0.00      0.00
10:54:18 AM      eth0     44.00     19.00     10.68     19.43      0.00      0.00      0.00

10:54:18 AM     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
10:54:19 AM        lo    276.24    276.24    126.01    126.01      0.00      0.00      0.00
10:54:19 AM      eth0     33.66     14.85      7.01     17.19      0.00      0.00      0.00

10:54:19 AM     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
10:54:20 AM        lo     67.01     67.01     27.14     27.14      0.00      0.00      0.00
10:54:20 AM      eth0     42.27     22.68      9.07     24.22      0.00      0.00      0.00

10:54:20 AM     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
10:54:21 AM        lo     22.22     22.22      4.95      4.95      0.00      0.00      0.00
10:54:21 AM      eth0     53.54     32.32      6.08     24.35      0.00      0.00      0.00

10:54:21 AM     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
10:54:22 AM        lo     83.00     83.00     17.89     17.89      0.00      0.00      0.00
10:54:22 AM      eth0     72.00     47.00     14.69      5.85      0.00      0.00      0.00

10:54:22 AM     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
10:54:23 AM        lo     22.22     22.22     14.77     14.77      0.00      0.00      0.00
10:54:23 AM      eth0     51.52     22.22     12.38     13.98      0.00      0.00      0.00

Write with dd seem to be at a reasonable speed:

[root@m10pmds autoit]# dd if=/dev/zero of=/root/test.img count=1000
1000+0 records in
1000+0 records out
512000 bytes (512 kB) copied, 0.00214131 s, 239 MB/s

dmesg output:

session_init(mms,17300): OK. kdev=fd:1, bs=4096.
register_make_request(mms,17300): OK. kdev=fd:1, mq=0.
session_init(mms,17300): OK. kdev=fd:1, bs=4096.
register_make_request(mms,17300): OK. kdev=fd:1, mq=0.
session_init(mms,19415): OK. kdev=fd:1, bs=4096.
register_make_request(mms,19415): OK. kdev=fd:1, mq=0.
session_init(service_process,32258): OK. kdev=8:3, bs=4096.
register_make_request(service_process,32258): OK. kdev=8:3, mq=0.
session_init(service_process,32258): OK. kdev=fd:1, bs=4096.
register_make_request(service_process,32258): OK. kdev=fd:1, mq=0.
session_init(service_process,32258): OK. kdev=fd:1, bs=4096.
register_make_request(service_process,32258): OK. kdev=fd:1, mq=0.
session_init(service_process,32258): OK. kdev=fd:1, bs=4096.
register_make_request(service_process,32258): OK. kdev=fd:1, mq=0.
session_init(service_process,32258): OK. kdev=fd:1, bs=4096.
register_make_request(service_process,32258): OK. kdev=fd:1, mq=0.
session_init(service_process,32258): OK. kdev=fd:1, bs=4096.
register_make_request(service_process,32258): OK. kdev=fd:1, mq=0.
session_init(mms,19462): OK. kdev=fd:1, bs=4096.
register_make_request(mms,19462): OK. kdev=fd:1, mq=0.
session_init(service_process,32258): OK. kdev=fd:1, bs=4096.
register_make_request(service_process,32258): OK. kdev=fd:1, mq=0.
session_init(service_process,32258): OK. kdev=fd:1, bs=4096.
register_make_request(service_process,32258): OK. kdev=fd:1, mq=0.
session_init(service_process,32258): OK. kdev=fd:1, bs=4096.
register_make_request(service_process,32258): OK. kdev=fd:1, mq=0.
session_init(service_process,32258): OK. kdev=fd:1, bs=4096.
register_make_request(service_process,32258): OK. kdev=fd:1, mq=0.
session_init(service_process,32258): OK. kdev=fd:1, bs=4096.
register_make_request(service_process,32258): OK. kdev=fd:1, mq=0.
session_init(service_process,32258): OK. kdev=fd:1, bs=4096.
register_make_request(service_process,32258): OK. kdev=fd:1, mq=0.
session_init(service_process,32258): OK. kdev=fd:1, bs=4096.
register_make_request(service_process,32258): OK. kdev=fd:1, mq=0.
session_init(service_process,32258): OK. kdev=fd:1, bs=4096.
register_make_request(service_process,32258): OK. kdev=fd:1, mq=0.
session_init(service_process,32258): OK. kdev=fd:1, bs=4096.
register_make_request(service_process,32258): OK. kdev=fd:1, mq=0.
session_init(service_process,32258): OK. kdev=fd:1, bs=4096.
register_make_request(service_process,32258): OK. kdev=fd:1, mq=0.
sa_cache_emlist_init(service_process,32258): Note, continue with 7436 pages.
session_init(service_process,32258): OK. kdev=8:1, bs=512.
register_make_request(service_process,32258): OK. kdev=8:1, mq=0.
session_stat(service_process,32258): dev=8:1 state=6 blksize=512 mmapsize=262144
session_stat(service_process,32258): psize=409600 pstrt=2048 mshft=0 ioctls=21
session_stat(service_process,32258): bhpgs=0 bhcnt=0 abhs=8 fbhs=8 dbhs=0
session_stat(service_process,32258): gpgs=7568 ppgs=7568 emmax=10610 emmin=9947 emcur=0 cached=0
session_stat(service_process,32258): rblk=497 cblk=0 rcblk=0 rc2blk=0 mcblk=0 rwcolls=0
session_stat(service_process,32258): sync=0 async=0 aretr=0 mipr=0 iprcnt=0
session_stat(service_process,32258): mbio=0 ioctlcnt=21 ioctlpid=20
session_stat(service_process,32258): rccalls=497 maxrcdepth=0 rcdepthcnts=(0, 0, 0, 0)
session_stat(service_process,32258): flags=0
session_init(service_process,32258): OK. kdev=8:2, bs=1024.
register_make_request(service_process,32258): OK. kdev=8:2, mq=0.
session_stat(service_process,32258): dev=8:2 state=6 blksize=1024 mmapsize=262144
session_stat(service_process,32258): psize=1024000 pstrt=411648 mshft=0 ioctls=586
session_stat(service_process,32258): bhpgs=0 bhcnt=0 abhs=565 fbhs=565 dbhs=0
session_stat(service_process,32258): gpgs=10774 ppgs=10774 emmax=10610 emmin=9947 emcur=0 cached=0
session_stat(service_process,32258): rblk=9303 cblk=73 rcblk=44 rc2blk=0 mcblk=35 rwcolls=5
session_stat(service_process,32258): sync=0 async=19 aretr=0 mipr=0 iprcnt=0
session_stat(service_process,32258): mbio=0 ioctlcnt=586 ioctlpid=585
session_stat(service_process,32258): rccalls=9407 maxrcdepth=0 rcdepthcnts=(0, 0, 0, 0)
session_stat(service_process,32258): flags=0
session_init(service_process,32258): OK. kdev=fd:5, bs=4096.
register_make_request(service_process,32258): OK. kdev=fd:5, mq=0.
session_stat(service_process,32258): dev=fd:5 state=6 blksize=4096 mmapsize=262144
session_stat(service_process,32258): psize=10485760 pstrt=0 mshft=0 ioctls=2720
session_stat(service_process,32258): bhpgs=0 bhcnt=0 abhs=1734 fbhs=1734 dbhs=0
session_stat(service_process,32258): gpgs=10816 ppgs=10816 emmax=10610 emmin=9947 emcur=0 cached=0
session_stat(service_process,32258): rblk=46749 cblk=41 rcblk=42 rc2blk=0 mcblk=40 rwcolls=2
session_stat(service_process,32258): sync=0 async=41 aretr=0 mipr=0 iprcnt=0
session_stat(service_process,32258): mbio=0 ioctlcnt=2720 ioctlpid=2719
session_stat(service_process,32258): rccalls=46829 maxrcdepth=0 rcdepthcnts=(0, 0, 0, 0)
session_stat(service_process,32258): flags=0
session_init(service_process,32258): OK. kdev=fd:3, bs=4096.
register_make_request(service_process,32258): OK. kdev=fd:3, mq=0.
session_stat(service_process,32258): dev=fd:3 state=6 blksize=4096 mmapsize=262144
session_stat(service_process,32258): psize=52428800 pstrt=0 mshft=0 ioctls=4448
session_stat(service_process,32258): bhpgs=0 bhcnt=0 abhs=3939 fbhs=3939 dbhs=0
session_stat(service_process,32258): gpgs=11142 ppgs=11142 emmax=10610 emmin=9947 emcur=0 cached=0
session_stat(service_process,32258): rblk=113418 cblk=201 rcblk=202 rc2blk=0 mcblk=200 rwcolls=2
session_stat(service_process,32258): sync=0 async=201 aretr=0 mipr=0 iprcnt=0
session_stat(service_process,32258): mbio=0 ioctlcnt=4448 ioctlpid=4447
session_stat(service_process,32258): rccalls=113818 maxrcdepth=1 rcdepthcnts=(0, 0, 0, 0)
session_stat(service_process,32258): flags=0
session_init(service_process,32258): OK. kdev=fd:2, bs=4096.
register_make_request(service_process,32258): OK. kdev=fd:2, mq=0.
session_stat(service_process,32258): dev=fd:2 state=6 blksize=4096 mmapsize=262144
session_stat(service_process,32258): psize=136314880 pstrt=0 mshft=0 ioctls=4039
session_stat(service_process,32258): bhpgs=0 bhcnt=0 abhs=3722 fbhs=3722 dbhs=0
session_stat(service_process,32258): gpgs=11787 ppgs=11787 emmax=10610 emmin=9947 emcur=0 cached=0
session_stat(service_process,32258): rblk=101949 cblk=521 rcblk=522 rc2blk=0 mcblk=520 rwcolls=2
session_stat(service_process,32258): sync=0 async=521 aretr=0 mipr=0 iprcnt=0
session_stat(service_process,32258): mbio=0 ioctlcnt=4039 ioctlpid=4038
session_stat(service_process,32258): rccalls=102989 maxrcdepth=0 rcdepthcnts=(0, 0, 0, 0)
session_stat(service_process,32258): flags=0
session_init(service_process,32258): OK. kdev=fd:0, bs=4096.
register_make_request(service_process,32258): OK. kdev=fd:0, mq=0.
session_stat(service_process,32258): dev=fd:0 state=6 blksize=4096 mmapsize=262144
session_stat(service_process,32258): psize=41943040 pstrt=0 mshft=0 ioctls=18387
session_stat(service_process,32258): bhpgs=0 bhcnt=0 abhs=11746 fbhs=11746 dbhs=0
session_stat(service_process,32258): gpgs=11077 ppgs=11077 emmax=10610 emmin=9947 emcur=0 cached=0
session_stat(service_process,32258): rblk=500297 cblk=178 rcblk=179 rc2blk=0 mcblk=160 rwcolls=22
session_stat(service_process,32258): sync=0 async=178 aretr=0 mipr=0 iprcnt=0
session_stat(service_process,32258): mbio=0 ioctlcnt=18387 ioctlpid=18386
session_stat(service_process,32258): rccalls=500617 maxrcdepth=1 rcdepthcnts=(0, 0, 0, 0)
session_stat(service_process,32258): flags=0
session_init(mms,19418): OK. kdev=fd:1, bs=4096.
register_make_request(mms,19418): OK. kdev=fd:1, mq=0.
snumbdctl_open(mms,19420): OK s=ffff88013de59000
session_init(mms,19420): OK. kdev=fc:1, len=46137344 sect=512 s=ffff88013de59000 pgrp=(15482).
snumbd_open_blk(blkid,32480): Disable access (556,15482,15482)...
snumbd_open_blk(udisks-part-id,32481): Disable access (556,15482,15482)...
EXT4-fs (snumbd1d): ext4_check_descriptors: Checksum for group 128 failed (22619!=896)
EXT4-fs (snumbd1d): group descriptors corrupted!
snumbdctl_release(mms,19420): OK s=ffff88013de59000
snumbdctl_open(mms,19420): OK s=ffff88013de59000
session_init(mms,19420): OK. kdev=fc:1, len=16351232 sect=512 s=ffff88013de59000 pgrp=(15482).
snumbd_open_blk(blkid,32489): Disable access (556,15482,15482)...
snumbd_open_blk(udisks-part-id,32490): Disable access (556,15482,15482)...
snumbdctl_release(mms,19420): OK s=ffff88013de59000
storvsc: Sense Key : Unit Attention [current]
storvsc: Add. Sense: Changed operating definition
sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters.
storvsc: Sense Key : Unit Attention [current]
storvsc: Add. Sense: Changed operating definition
sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters.
4
  • Is there anything IO related in dmesg? Write with dd seem to be at a reasonable speed - you don't know that, you've not run sync after it: stackoverflow.com/questions/49489798/… Oct 31, 2022 at 2:49
  • I don't notice any slowness with sync. I've updated my post and added the dmesg logs. There is a group descriptors corrupted which I am asking Acronis to assist with.
    – supmethods
    Oct 31, 2022 at 15:31
  • EXT4-fs (snumbd1d): ext4_check_descriptors: Checksum for group 128 failed (22619!=896) EXT4-fs (snumbd1d): group descriptors corrupted! - this doesn't look good. I've no idea what storage it is but it should absolutely not happen. At the very least unmount and run e2fsck -f. You need to check if your storage is OK. I was about to say something about showing smartctl -a /dev/device but I don't whether your storage is even SMART compliant. Oct 31, 2022 at 15:39
  • I took the system offline and ran fsck on all the partition with a file system but no error was reported. I am asking Acronis about the group descriptors corrupted as it is their SnapAPI that's reporting the error. I have asked Fujitsu for a way to obtained the SMART information of hard drives. The software I tried to use couldn't obtain the SMART information because it is behind RAID.
    – supmethods
    Oct 31, 2022 at 15:48

1 Answer 1

0

Hope you're not still waiting on a solution, but I'll share this: Whenever my systems go into a high load state, without the process load adding up, it is always due to an NFS mount gone bad. I umount -lf them all and the load comes down right away. Then I try to mount -a them again and the culprit will show: mount.nfs: backgrounding "system:share"

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .