I'm running in a severe performance issue with the following setup and was hoping for some insights how to debug them. I run a level 6 dm-raid RAID with 4 HDDs and started replacing the RAID partitions (sda1, sdb1, sdc1, sdd1) one by one with dm-integrity volumes (dm-0, dm-1, dm-2, dm-3):
/dev/md0 /dev/sda1 → /dev/dm-0 /dev/sdb1 → /dev/dm-1 /dev/sdc1 → tbd /dev/sdd1 → tbd
The first replacement (sda1→dm-0) went fine, RAID resync went with around 100MB/s. However the second replacement (sdb1→dm-1) averaged at 10MB/s for two hours before I stopped the sync. I reverted the last replacement (dm1→sdb1) and noticed that resync was about 60MB/s. Just for the fun of it I failed dm-0 from the RAID because the remaining sdc1 and sdd1 should be enough for the resync and the speed went up to 100MB/s. At all times the CPU usage doesn't go over 30%.
Things I tried that don't affect speed: integritysetup sector size (tried both 512 and 4096) and checksum algorithm (tried crc32c and sha1), hdparm read-ahead.
The CPU is a Xeon E3-1220L V2 @ 2.30GHz with hardware support for encryption. lscpu flags:
fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm cpuid_fault epb pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts md_clear flush_l1d