3

I rebuilt a machine recently, and decided it would be fun to do RAID on some SDDs. It has not been going well, and I'm having trouble figuring out what is going wrong. I have 2 8TB HDDs in RAID 1 and those are working fine. I also have 4 1TB SDDs in RAID 10 and that's where the problem is. Everything seems fine when the system first boots. However some hours later, performance degrades and I start seeing errors.

I'll post some basic configuration and logs of what I'm seeing. Please let me know if there is anything else I should provide.

cat /proc/mdstat

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md1 : active raid1 sde2[0]
      244008960 blocks super 1.2 [2/1] [U_]
      bitmap: 2/2 pages [8KB], 65536KB chunk

md2 : active raid1 sde3[0]
      7568776192 blocks super 1.2 [2/1] [U_]
      bitmap: 4/57 pages [16KB], 65536KB chunk

md0 : active raid1 sde1[0]
      974848 blocks super 1.2 [2/1] [U_]

md3 : active raid10 sda1[0] sdb1[1] sdc1[2]
      1953257472 blocks super 1.2 512K chunks 2 near-copies [4/3] [UUU_]
      bitmap: 2/15 pages [8KB], 65536KB chunk

unused devices: <none>

Note that /dev/sdd1 seems to be missing from md3.

lsblk

NAME    MAJ:MIN RM   SIZE RO TYPE   MOUNTPOINT
loop0     7:0    0     4G  0 loop   
loop1     7:1    0    20G  0 loop   
loop2     7:2    0     4G  0 loop   
loop3     7:3    0   512M  0 loop   
loop4     7:4    0   3.9T  0 loop   
loop5     7:5    0     4G  0 loop   
loop6     7:6    0    20G  0 loop   
loop7     7:7    0     2G  0 loop   
sda       8:0    0 931.5G  0 disk   
└─sda1    8:1    0 931.5G  0 part   
  └─md3   9:3    0   1.8T  0 raid10 /perf
sdb       8:16   0 931.5G  0 disk   
└─sdb1    8:17   0 931.5G  0 part   
  └─md3   9:3    0   1.8T  0 raid10 /perf
sdc       8:32   0 931.5G  0 disk   
└─sdc1    8:33   0 931.5G  0 part   
  └─md3   9:3    0   1.8T  0 raid10 /perf
sdd       8:48   0 931.5G  0 disk   
└─sdd1    8:49   0 931.5G  0 part   
sde       8:64   0   7.3T  0 disk   
├─sde1    8:65   0   953M  0 part   
│ └─md0   9:0    0   952M  0 raid1  /boot
├─sde2    8:66   0 232.9G  0 part   
│ └─md1   9:1    0 232.7G  0 raid1  /
└─sde3    8:67   0   7.1T  0 part   
  └─md2   9:2    0   7.1T  0 raid1  /bulk
sdf       8:80   0   7.3T  0 disk   
├─sdf1    8:81   0   953M  0 part   
├─sdf2    8:82   0 232.9G  0 part   
└─sdf3    8:83   0   7.1T  0 part  

Seems like sdd1 has also gone missing from md3 here.

Here are some dmesg logs:

sudo dmesg | grep -e "ata[1-4]"

[    5.134388] ata1: SATA max UDMA/133 abar m1024@0xfe40b000 port 0xfe40b100 irq 19
[    5.134390] ata2: SATA max UDMA/133 abar m1024@0xfe40b000 port 0xfe40b180 irq 19
[    5.134392] ata3: SATA max UDMA/133 abar m1024@0xfe40b000 port 0xfe40b200 irq 19
[    5.134394] ata4: SATA max UDMA/133 abar m1024@0xfe40b000 port 0xfe40b280 irq 19
[    5.607811] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    5.607831] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    5.608046] ata1.00: supports DRM functions and may not be fully accessible
[    5.608067] ata2.00: supports DRM functions and may not be fully accessible
[    5.608482] ata1.00: ATA-11: Samsung SSD 860 QVO 1TB, RVQ01B6Q, max UDMA/133
[    5.608483] ata1.00: 1953525168 sectors, multi 1: LBA48 NCQ (depth 32), AA
[    5.608507] ata2.00: ATA-11: Samsung SSD 860 QVO 1TB, RVQ01B6Q, max UDMA/133
[    5.608508] ata2.00: 1953525168 sectors, multi 1: LBA48 NCQ (depth 32), AA
[    5.610194] ata1.00: supports DRM functions and may not be fully accessible
[    5.610217] ata2.00: supports DRM functions and may not be fully accessible
[    5.611835] ata3: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    5.612033] ata1.00: configured for UDMA/133
[    5.612044] ata2.00: configured for UDMA/133
[    5.612072] ata3.00: supports DRM functions and may not be fully accessible
[    5.612555] ata3.00: ATA-11: Samsung SSD 860 QVO 1TB, RVQ01B6Q, max UDMA/133
[    5.612557] ata3.00: 1953525168 sectors, multi 1: LBA48 NCQ (depth 32), AA
[    5.614228] ata3.00: supports DRM functions and may not be fully accessible
[    5.615223] ata1.00: Enabling discard_zeroes_data
[    5.615253] ata2.00: Enabling discard_zeroes_data
[    5.615391] ata1.00: Enabling discard_zeroes_data
[    5.615428] ata2.00: Enabling discard_zeroes_data
[    5.615809] ata4: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    5.616059] ata3.00: configured for UDMA/133
[    5.616069] ata4.00: supports DRM functions and may not be fully accessible
[    5.616083] ata2.00: Enabling discard_zeroes_data
[    5.616147] ata1.00: Enabling discard_zeroes_data
[    5.616308] ata3.00: Enabling discard_zeroes_data
[    5.616444] ata3.00: Enabling discard_zeroes_data
[    5.616518] ata4.00: ATA-11: Samsung SSD 860 QVO 1TB, RVQ01B6Q, max UDMA/133
[    5.616520] ata4.00: 1953525168 sectors, multi 1: LBA48 NCQ (depth 32), AA
[    5.617064] ata3.00: Enabling discard_zeroes_data
[    5.618187] ata4.00: supports DRM functions and may not be fully accessible
[    5.619969] ata4.00: configured for UDMA/133
[    5.620204] ata4.00: Enabling discard_zeroes_data
[    5.620336] ata4.00: Enabling discard_zeroes_data
[    5.620905] ata4.00: Enabling discard_zeroes_data
[ 4675.428207] ata3: log page 10h reported inactive tag 26
[ 4675.428226] ata3.00: exception Emask 0x1 SAct 0x78000003 SErr 0x0 action 0x0
[ 4675.428232] ata3.00: irq_stat 0x40000008
[ 4675.428237] ata3.00: failed command: WRITE FPDMA QUEUED
[ 4675.428244] ata3.00: cmd 61/10:00:c0:96:c7/00:00:00:00:00/40 tag 0 ncq dma 8192 out
[ 4675.428255] ata3.00: status: { DRDY }
[ 4675.428259] ata3.00: failed command: WRITE FPDMA QUEUED
[ 4675.428266] ata3.00: cmd 61/10:08:70:dd:c7/00:00:00:00:00/40 tag 1 ncq dma 8192 out
[ 4675.428276] ata3.00: status: { DRDY }
[ 4675.428280] ata3.00: failed command: WRITE FPDMA QUEUED
[ 4675.428286] ata3.00: cmd 61/08:d8:00:29:b8/00:00:00:00:00/40 tag 27 ncq dma 4096 out
[ 4675.428296] ata3.00: status: { DRDY }
[ 4675.428300] ata3.00: failed command: WRITE FPDMA QUEUED
[ 4675.428307] ata3.00: cmd 61/08:e0:60:1e:be/00:00:00:00:00/40 tag 28 ncq dma 4096 out
[ 4675.428316] ata3.00: status: { DRDY }
[ 4675.428320] ata3.00: failed command: WRITE FPDMA QUEUED
[ 4675.428326] ata3.00: cmd 61/10:e8:10:c4:c2/00:00:00:00:00/40 tag 29 ncq dma 8192 out
[ 4675.428336] ata3.00: status: { DRDY }
[ 4675.428343] ata3.00: failed command: WRITE FPDMA QUEUED
[ 4675.428350] ata3.00: cmd 61/08:f0:78:cd:c6/00:00:00:00:00/40 tag 30 ncq dma 4096 out
[ 4675.428359] ata3.00: status: { DRDY }
[ 4675.428584] ata3.00: supports DRM functions and may not be fully accessible
[ 4675.430661] ata3.00: supports DRM functions and may not be fully accessible
[ 4675.432473] ata3.00: configured for UDMA/133
[ 4675.432485] ata3: EH complete
[ 4675.432522] ata3.00: Enabling discard_zeroes_data
[ 7924.795947] ata3: log page 10h reported inactive tag 13
[ 7924.795964] ata3.00: exception Emask 0x1 SAct 0x1c000 SErr 0x0 action 0x0
[ 7924.795970] ata3.00: irq_stat 0x40000008
[ 7924.795976] ata3.00: failed command: WRITE FPDMA QUEUED
[ 7924.795983] ata3.00: cmd 61/08:70:30:c9:23/00:00:00:00:00/40 tag 14 ncq dma 4096 out
[ 7924.795993] ata3.00: status: { DRDY }
[ 7924.795998] ata3.00: failed command: WRITE FPDMA QUEUED
[ 7924.796004] ata3.00: cmd 61/08:78:18:ba:ab/00:00:00:00:00/40 tag 15 ncq dma 4096 out
[ 7924.796014] ata3.00: status: { DRDY }
[ 7924.796018] ata3.00: failed command: WRITE FPDMA QUEUED
[ 7924.796025] ata3.00: cmd 61/08:80:a8:9b:57/00:00:00:00:00/40 tag 16 ncq dma 4096 out
[ 7924.796034] ata3.00: status: { DRDY }
[ 7924.796264] ata3.00: supports DRM functions and may not be fully accessible
[ 7924.798340] ata3.00: supports DRM functions and may not be fully accessible
[ 7924.800138] ata3.00: configured for UDMA/133
[ 7924.800149] ata3: EH complete
[ 7924.800232] ata3.00: Enabling discard_zeroes_data
[14346.389219] ata2: log page 10h reported inactive tag 23
[14346.389239] ata2.00: exception Emask 0x1 SAct 0x1000000 SErr 0x0 action 0x0
[14346.389245] ata2.00: irq_stat 0x40000008
[14346.389250] ata2.00: failed command: WRITE FPDMA QUEUED
[14346.389258] ata2.00: cmd 61/10:c0:70:3c:4e/00:00:00:00:00/40 tag 24 ncq dma 8192 out
[14346.389268] ata2.00: status: { DRDY }
[14346.389494] ata2.00: supports DRM functions and may not be fully accessible
[14346.391576] ata2.00: supports DRM functions and may not be fully accessible
[14346.393430] ata2.00: configured for UDMA/133
[14346.393439] ata2: EH complete
[14346.393475] ata2.00: Enabling discard_zeroes_data
[15627.956931] ata3: log page 10h reported inactive tag 13
[15627.956945] ata3.00: exception Emask 0x1 SAct 0x7c000 SErr 0x0 action 0x0
[15627.956951] ata3.00: irq_stat 0x40000008
[15627.956956] ata3.00: failed command: WRITE FPDMA QUEUED
[15627.957135] ata3.00: cmd 61/08:70:a8:9b:57/00:00:00:00:00/40 tag 14 ncq dma 4096 out
[15627.958031] ata3.00: status: { DRDY }
[15627.958628] ata3.00: failed command: WRITE FPDMA QUEUED
[15627.959221] ata3.00: cmd 61/08:78:40:29:b8/00:00:00:00:00/40 tag 15 ncq dma 4096 out
[15627.960442] ata3.00: status: { DRDY }
[15627.961017] ata3.00: failed command: WRITE FPDMA QUEUED
[15627.961613] ata3.00: cmd 61/08:80:e8:c6:c2/00:00:00:00:00/40 tag 16 ncq dma 4096 out
[15627.962819] ata3.00: status: { DRDY }
[15627.963417] ata3.00: failed command: WRITE FPDMA QUEUED
[15627.964019] ata3.00: cmd 61/10:88:a0:8a:c7/00:00:00:00:00/40 tag 17 ncq dma 8192 out
[15627.965239] ata3.00: status: { DRDY }
[15627.965826] ata3.00: failed command: WRITE FPDMA QUEUED
[15627.966426] ata3.00: cmd 61/10:90:98:e2:c7/00:00:00:00:00/40 tag 18 ncq dma 8192 out
[15627.967634] ata3.00: status: { DRDY }
[15627.968458] ata3.00: supports DRM functions and may not be fully accessible
[15627.970477] ata3.00: supports DRM functions and may not be fully accessible
[15627.972244] ata3.00: configured for UDMA/133
[15627.972251] ata3: EH complete
[15627.972305] ata3.00: Enabling discard_zeroes_data
[16342.275376] ata1.00: exception Emask 0x10 SAct 0x70000007 SErr 0x0 action 0x6 frozen
[16342.275613] ata1.00: irq_stat 0x08000000, interface fatal error
[16342.275902] ata1.00: failed command: WRITE FPDMA QUEUED
[16342.276487] ata1.00: cmd 61/08:00:20:20:be/00:00:00:00:00/40 tag 0 ncq dma 4096 out
[16342.277708] ata1.00: status: { DRDY }
[16342.278333] ata1.00: failed command: WRITE FPDMA QUEUED
[16342.278922] ata1.00: cmd 61/10:08:c8:49:c6/00:00:00:00:00/40 tag 1 ncq dma 8192 out
[16342.280186] ata1.00: status: { DRDY }
[16342.280774] ata1.00: failed command: WRITE FPDMA QUEUED
[16342.281366] ata1.00: cmd 61/08:10:e0:53:c7/00:00:00:00:00/40 tag 2 ncq dma 4096 out
[16342.282600] ata1.00: status: { DRDY }
[16342.283216] ata1.00: failed command: WRITE FPDMA QUEUED
[16342.283853] ata1.00: cmd 61/10:e0:c0:e1:9c/00:00:00:00:00/40 tag 28 ncq dma 8192 out
[16342.285064] ata1.00: status: { DRDY }
[16342.285677] ata1.00: failed command: WRITE FPDMA QUEUED
[16342.286292] ata1.00: cmd 61/10:e8:68:e2:9c/00:00:00:00:00/40 tag 29 ncq dma 8192 out
[16342.287542] ata1.00: status: { DRDY }
[16342.288147] ata1.00: failed command: WRITE FPDMA QUEUED
[16342.288760] ata1.00: cmd 61/28:f0:80:e2:9c/00:00:00:00:00/40 tag 30 ncq dma 20480 out
[16342.290003] ata1.00: status: { DRDY }
[16342.290622] ata1: hard resetting link
[16342.767375] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[16342.767614] ata1.00: supports DRM functions and may not be fully accessible
[16342.769754] ata1.00: supports DRM functions and may not be fully accessible
[16342.771578] ata1.00: configured for UDMA/133
[16342.771592] ata1: EH complete
[16342.771647] ata1.00: Enabling discard_zeroes_data
[18043.179405] ata1: log page 10h reported inactive tag 15
[18043.179691] ata1.00: exception Emask 0x1 SAct 0x3f0000 SErr 0x0 action 0x0
[18043.179936] ata1.00: irq_stat 0x40000008
[18043.180546] ata1.00: failed command: WRITE FPDMA QUEUED
[18043.181163] ata1.00: cmd 61/08:80:60:cc:23/00:00:00:00:00/40 tag 16 ncq dma 4096 out
[18043.182410] ata1.00: status: { DRDY }
[18043.183061] ata1.00: failed command: WRITE FPDMA QUEUED
[18043.183648] ata1.00: cmd 61/08:88:98:ce:23/00:00:00:00:00/40 tag 17 ncq dma 4096 out
[18043.184901] ata1.00: status: { DRDY }
[18043.185523] ata1.00: failed command: WRITE FPDMA QUEUED
[18043.186146] ata1.00: cmd 61/10:90:70:3c:4e/00:00:00:00:00/40 tag 18 ncq dma 8192 out
[18043.187423] ata1.00: status: { DRDY }
[18043.188029] ata1.00: failed command: WRITE FPDMA QUEUED
[18043.188649] ata1.00: cmd 61/08:98:58:90:57/00:00:00:00:00/40 tag 19 ncq dma 4096 out
[18043.189910] ata1.00: status: { DRDY }
[18043.190534] ata1.00: failed command: WRITE FPDMA QUEUED
[18043.191174] ata1.00: cmd 61/10:a0:a0:8c:c7/00:00:00:00:00/40 tag 20 ncq dma 8192 out
[18043.192414] ata1.00: status: { DRDY }
[18043.193044] ata1.00: failed command: WRITE FPDMA QUEUED
[18043.193657] ata1.00: cmd 61/10:a8:98:e4:c7/00:00:00:00:00/40 tag 21 ncq dma 8192 out
[18043.194929] ata1.00: status: { DRDY }
[18043.195775] ata1.00: supports DRM functions and may not be fully accessible
[18043.197860] ata1.00: supports DRM functions and may not be fully accessible
[18043.199682] ata1.00: configured for UDMA/133
[18043.199695] ata1: EH complete
[18043.199734] ata1.00: Enabling discard_zeroes_data
[21426.517162] ata1: log page 10h reported inactive tag 26
[21426.517358] ata1.00: exception Emask 0x1 SAct 0x78000007 SErr 0x0 action 0x0
[21426.517707] ata1.00: irq_stat 0x40000008
[21426.518302] ata1.00: failed command: WRITE FPDMA QUEUED
[21426.518930] ata1.00: cmd 61/10:00:68:e2:9c/00:00:00:00:00/40 tag 0 ncq dma 8192 out
[21426.520164] ata1.00: status: { DRDY }
[21426.520803] ata1.00: failed command: WRITE FPDMA QUEUED
[21426.521397] ata1.00: cmd 61/40:08:80:e2:9c/00:00:00:00:00/40 tag 1 ncq dma 32768 out
[21426.522641] ata1.00: status: { DRDY }
[21426.523257] ata1.00: failed command: WRITE FPDMA QUEUED
[21426.523870] ata1.00: cmd 61/08:10:88:9d:c7/00:00:00:00:00/40 tag 2 ncq dma 4096 out
[21426.525126] ata1.00: status: { DRDY }
[21426.525725] ata1.00: failed command: WRITE FPDMA QUEUED
[21426.526338] ata1.00: cmd 61/08:d8:78:3c:4e/00:00:00:00:00/40 tag 27 ncq dma 4096 out
[21426.527573] ata1.00: status: { DRDY }
[21426.528190] ata1.00: failed command: WRITE FPDMA QUEUED
[21426.528816] ata1.00: cmd 61/08:e0:70:90:57/00:00:00:00:00/40 tag 28 ncq dma 4096 out
[21426.530042] ata1.00: status: { DRDY }
[21426.530657] ata1.00: failed command: WRITE FPDMA QUEUED
[21426.531270] ata1.00: cmd 61/08:e8:88:cd:57/00:00:00:00:00/40 tag 29 ncq dma 4096 out
[21426.532520] ata1.00: status: { DRDY }
[21426.533124] ata1.00: failed command: WRITE FPDMA QUEUED
[21426.533738] ata1.00: cmd 61/10:f0:c0:e1:9c/00:00:00:00:00/40 tag 30 ncq dma 8192 out
[21426.534971] ata1.00: status: { DRDY }
[21426.535804] ata1.00: supports DRM functions and may not be fully accessible
[21426.537908] ata1.00: supports DRM functions and may not be fully accessible
[21426.539726] ata1.00: configured for UDMA/133
[21426.539739] ata1: EH complete
[21426.539773] ata1.00: Enabling discard_zeroes_data
[26919.670548] ata1: log page 10h reported inactive tag 27
[26919.670742] ata1.00: exception Emask 0x1 SAct 0x7000000f SErr 0x0 action 0x0
[26919.671054] ata1.00: irq_stat 0x40000008
[26919.671673] ata1.00: failed command: WRITE FPDMA QUEUED
[26919.672287] ata1.00: cmd 61/10:00:c0:e1:9c/00:00:00:00:00/40 tag 0 ncq dma 8192 out
[26919.673522] ata1.00: status: { DRDY }
[26919.674167] ata1.00: failed command: WRITE FPDMA QUEUED
[26919.674752] ata1.00: cmd 61/10:08:68:e2:9c/00:00:00:00:00/40 tag 1 ncq dma 8192 out
[26919.675987] ata1.00: status: { DRDY }
[26919.676602] ata1.00: failed command: WRITE FPDMA QUEUED
[26919.677215] ata1.00: cmd 61/40:10:80:e2:9c/00:00:00:00:00/40 tag 2 ncq dma 32768 out
[26919.678467] ata1.00: status: { DRDY }
[26919.679063] ata1.00: failed command: WRITE FPDMA QUEUED
[26919.679680] ata1.00: cmd 61/10:18:10:b0:c6/00:00:00:00:00/40 tag 3 ncq dma 8192 out
[26919.680915] ata1.00: status: { DRDY }
[26919.681527] ata1.00: failed command: WRITE FPDMA QUEUED
[26919.682152] ata1.00: cmd 61/08:e0:98:ce:23/00:00:00:00:00/40 tag 28 ncq dma 4096 out
[26919.683373] ata1.00: status: { DRDY }
[26919.683991] ata1.00: failed command: WRITE FPDMA QUEUED
[26919.684602] ata1.00: cmd 61/10:e8:70:3c:4e/00:00:00:00:00/40 tag 29 ncq dma 8192 out
[26919.685855] ata1.00: status: { DRDY }
[26919.686452] ata1.00: failed command: WRITE FPDMA QUEUED
[26919.687067] ata1.00: cmd 61/08:f0:48:90:57/00:00:00:00:00/40 tag 30 ncq dma 4096 out
[26919.688313] ata1.00: status: { DRDY }
[26919.689162] ata1.00: supports DRM functions and may not be fully accessible
[26919.691243] ata1.00: supports DRM functions and may not be fully accessible
[26919.693047] ata1.00: configured for UDMA/133
[26919.693059] ata1: EH complete
[26919.693108] ata1.00: Enabling discard_zeroes_data
[32097.151933] ata1: log page 10h reported inactive tag 27
[32097.152129] ata1.00: exception Emask 0x1 SAct 0x7000003f SErr 0x0 action 0x0
[32097.152447] ata1.00: irq_stat 0x40000008
[32097.153064] ata1.00: failed command: WRITE FPDMA QUEUED
[32097.153679] ata1.00: cmd 61/08:00:60:cc:23/00:00:00:00:00/40 tag 0 ncq dma 4096 out
[32097.154921] ata1.00: status: { DRDY }
[32097.155563] ata1.00: failed command: WRITE FPDMA QUEUED
[32097.156155] ata1.00: cmd 61/08:08:98:ce:23/00:00:00:00:00/40 tag 1 ncq dma 4096 out
[32097.157402] ata1.00: status: { DRDY }
[32097.158017] ata1.00: failed command: WRITE FPDMA QUEUED
[32097.158630] ata1.00: cmd 61/08:10:70:3c:4e/00:00:00:00:00/40 tag 2 ncq dma 4096 out
[32097.159891] ata1.00: status: { DRDY }
[32097.160498] ata1.00: failed command: WRITE FPDMA QUEUED
[32097.161111] ata1.00: cmd 61/08:18:18:90:57/00:00:00:00:00/40 tag 3 ncq dma 4096 out
[32097.162359] ata1.00: status: { DRDY }
[32097.162975] ata1.00: failed command: WRITE FPDMA QUEUED
[32097.163599] ata1.00: cmd 61/08:20:58:90:57/00:00:00:00:00/40 tag 4 ncq dma 4096 out
[32097.164834] ata1.00: status: { DRDY }
[32097.165453] ata1.00: failed command: WRITE FPDMA QUEUED
[32097.166065] ata1.00: cmd 61/08:28:b8:0c:b1/00:00:00:00:00/40 tag 5 ncq dma 4096 out
[32097.167318] ata1.00: status: { DRDY }
[32097.167920] ata1.00: failed command: WRITE FPDMA QUEUED
[32097.168534] ata1.00: cmd 61/08:e0:e0:c9:23/00:00:00:00:00/40 tag 28 ncq dma 4096 out
[32097.169777] ata1.00: status: { DRDY }
[32097.170411] ata1.00: failed command: WRITE FPDMA QUEUED
[32097.171009] ata1.00: cmd 61/08:e8:48:ca:23/00:00:00:00:00/40 tag 29 ncq dma 4096 out
[32097.172269] ata1.00: status: { DRDY }
[32097.172872] ata1.00: failed command: WRITE FPDMA QUEUED
[32097.173490] ata1.00: cmd 61/08:f0:88:ca:23/00:00:00:00:00/40 tag 30 ncq dma 4096 out
[32097.174740] ata1.00: status: { DRDY }
[32097.175590] ata1.00: supports DRM functions and may not be fully accessible
[32097.177669] ata1.00: supports DRM functions and may not be fully accessible
[32097.179479] ata1.00: configured for UDMA/133
[32097.179493] ata1: EH complete
[32097.179532] ata1.00: Enabling discard_zeroes_data
[32943.729356] ata2: log page 10h reported inactive tag 5
[32943.729555] ata2.00: exception Emask 0x1 SAct 0x3fffc0 SErr 0x0 action 0x0
[32943.729887] ata2.00: irq_stat 0x40000008
[32943.730490] ata2.00: failed command: WRITE FPDMA QUEUED
[32943.731112] ata2.00: cmd 61/08:30:e0:c9:23/00:00:00:00:00/40 tag 6 ncq dma 4096 out
[32943.732351] ata2.00: status: { DRDY }
[32943.732989] ata2.00: failed command: WRITE FPDMA QUEUED
[32943.733583] ata2.00: cmd 61/08:38:48:ca:23/00:00:00:00:00/40 tag 7 ncq dma 4096 out
[32943.734830] ata2.00: status: { DRDY }
[32943.735441] ata2.00: failed command: WRITE FPDMA QUEUED
[32943.736052] ata2.00: cmd 61/08:40:88:ca:23/00:00:00:00:00/40 tag 8 ncq dma 4096 out
[32943.737312] ata2.00: status: { DRDY }
[32943.737910] ata2.00: failed command: WRITE FPDMA QUEUED
[32943.738522] ata2.00: cmd 61/08:48:60:cc:23/00:00:00:00:00/40 tag 9 ncq dma 4096 out
[32943.739759] ata2.00: status: { DRDY }
[32943.740371] ata2.00: failed command: WRITE FPDMA QUEUED
[32943.741002] ata2.00: cmd 61/08:50:98:ce:23/00:00:00:00:00/40 tag 10 ncq dma 4096 out
[32943.742233] ata2.00: status: { DRDY }
[32943.742848] ata2.00: failed command: WRITE FPDMA QUEUED
[32943.743460] ata2.00: cmd 61/08:58:78:3c:4e/00:00:00:00:00/40 tag 11 ncq dma 4096 out
[32943.744722] ata2.00: status: { DRDY }
[32943.745322] ata2.00: failed command: WRITE FPDMA QUEUED
[32943.745933] ata2.00: cmd 61/08:60:c0:05:8c/00:00:00:00:00/40 tag 12 ncq dma 4096 out
[32943.747175] ata2.00: status: { DRDY }
[32943.747794] ata2.00: failed command: WRITE FPDMA QUEUED
[32943.748406] ata2.00: cmd 61/10:68:c0:e1:9c/00:00:00:00:00/40 tag 13 ncq dma 8192 out
[32943.749665] ata2.00: status: { DRDY }
[32943.750267] ata2.00: failed command: WRITE FPDMA QUEUED
[32943.750884] ata2.00: cmd 61/10:70:68:e2:9c/00:00:00:00:00/40 tag 14 ncq dma 8192 out
[32943.752132] ata2.00: status: { DRDY }
[32943.752764] ata2.00: failed command: WRITE FPDMA QUEUED
[32943.753362] ata2.00: cmd 61/28:78:80:e2:9c/00:00:00:00:00/40 tag 15 ncq dma 20480 out
[32943.754610] ata2.00: status: { DRDY }
[32943.755229] ata2.00: failed command: WRITE FPDMA QUEUED
[32943.755850] ata2.00: cmd 61/f0:80:e8:10:22/00:00:00:00:00/40 tag 16 ncq dma 122880 out
[32943.757127] ata2.00: status: { DRDY }
[32943.757727] ata2.00: failed command: WRITE FPDMA QUEUED
[32943.758341] ata2.00: cmd 61/08:88:18:90:57/00:00:00:00:00/40 tag 17 ncq dma 4096 out
[32943.759594] ata2.00: status: { DRDY }
[32943.760210] ata2.00: failed command: WRITE FPDMA QUEUED
[32943.760842] ata2.00: cmd 61/08:90:80:5d:73/00:00:00:00:00/40 tag 18 ncq dma 4096 out
[32943.762077] ata2.00: status: { DRDY }
[32943.762722] ata2.00: failed command: WRITE FPDMA QUEUED
[32943.763310] ata2.00: cmd 61/08:98:38:0c:b1/00:00:00:00:00/40 tag 19 ncq dma 4096 out
[32943.764561] ata2.00: status: { DRDY }
[32943.765197] ata2.00: failed command: WRITE FPDMA QUEUED
[32943.765801] ata2.00: cmd 61/10:a0:98:22:be/00:00:00:00:00/40 tag 20 ncq dma 8192 out
[32943.767055] ata2.00: status: { DRDY }
[32943.767680] ata2.00: failed command: WRITE FPDMA QUEUED
[32943.768299] ata2.00: cmd 61/08:a8:60:cb:c2/00:00:00:00:00/40 tag 21 ncq dma 4096 out
[32943.769565] ata2.00: status: { DRDY }
[32943.770408] ata2.00: supports DRM functions and may not be fully accessible
[32943.772508] ata2.00: supports DRM functions and may not be fully accessible
[32943.774295] ata2.00: configured for UDMA/133
[32943.774321] ata2: EH complete
[32943.774391] ata2.00: Enabling discard_zeroes_data
[35466.450252] ata2: log page 10h reported inactive tag 20
[35466.450447] ata2.00: exception Emask 0x1 SAct 0x7fe00003 SErr 0x0 action 0x0
[35466.450788] ata2.00: irq_stat 0x40000008
[35466.451388] ata2.00: failed command: WRITE FPDMA QUEUED
[35466.452016] ata2.00: cmd 61/08:00:88:94:c0/00:00:00:00:00/40 tag 0 ncq dma 4096 out
[35466.453261] ata2.00: status: { DRDY }
[35466.453921] ata2.00: failed command: WRITE FPDMA QUEUED
[35466.454505] ata2.00: cmd 61/10:08:98:a2:c7/00:00:00:00:00/40 tag 1 ncq dma 8192 out
[35466.455751] ata2.00: status: { DRDY }
[35466.456364] ata2.00: failed command: WRITE FPDMA QUEUED
[35466.456987] ata2.00: cmd 61/08:a8:60:9c:23/00:00:00:00:00/40 tag 21 ncq dma 4096 out
[35466.458246] ata2.00: status: { DRDY }
[35466.458845] ata2.00: failed command: WRITE FPDMA QUEUED
[35466.459462] ata2.00: cmd 61/08:b0:a0:9c:23/00:00:00:00:00/40 tag 22 ncq dma 4096 out
[35466.460706] ata2.00: status: { DRDY }
[35466.461324] ata2.00: failed command: WRITE FPDMA QUEUED
[35466.461958] ata2.00: cmd 61/10:b8:b8:9c:23/00:00:00:00:00/40 tag 23 ncq dma 8192 out
[35466.463189] ata2.00: status: { DRDY }
[35466.463808] ata2.00: failed command: WRITE FPDMA QUEUED
[35466.464424] ata2.00: cmd 61/08:c0:f8:9c:23/00:00:00:00:00/40 tag 24 ncq dma 4096 out
[35466.465685] ata2.00: status: { DRDY }
[35466.466288] ata2.00: failed command: WRITE FPDMA QUEUED
[35466.466908] ata2.00: cmd 61/08:c8:e0:c9:23/00:00:00:00:00/40 tag 25 ncq dma 4096 out
[35466.468152] ata2.00: status: { DRDY }
[35466.468770] ata2.00: failed command: WRITE FPDMA QUEUED
[35466.469388] ata2.00: cmd 61/08:d0:20:90:57/00:00:00:00:00/40 tag 26 ncq dma 4096 out
[35466.470665] ata2.00: status: { DRDY }
[35466.471252] ata2.00: failed command: WRITE FPDMA QUEUED
[35466.471871] ata2.00: cmd 61/08:d8:48:90:57/00:00:00:00:00/40 tag 27 ncq dma 4096 out
[35466.473114] ata2.00: status: { DRDY }
[35466.473745] ata2.00: failed command: WRITE FPDMA QUEUED
[35466.474350] ata2.00: cmd 61/10:e0:c0:e1:9c/00:00:00:00:00/40 tag 28 ncq dma 8192 out
[35466.475593] ata2.00: status: { DRDY }
[35466.476210] ata2.00: failed command: WRITE FPDMA QUEUED
[35466.476827] ata2.00: cmd 61/10:e8:68:e2:9c/00:00:00:00:00/40 tag 29 ncq dma 8192 out
[35466.478089] ata2.00: status: { DRDY }
[35466.478688] ata2.00: failed command: WRITE FPDMA QUEUED
[35466.479305] ata2.00: cmd 61/18:f0:80:e2:9c/00:00:00:00:00/40 tag 30 ncq dma 12288 out
[35466.480549] ata2.00: status: { DRDY }
[35466.481412] ata2.00: supports DRM functions and may not be fully accessible
[35466.483449] ata2.00: supports DRM functions and may not be fully accessible
[35466.485216] ata2.00: configured for UDMA/133
[35466.485232] ata2: EH complete
[35466.485273] ata2.00: Enabling discard_zeroes_data

Has /dev/sdd gone bad? Is there any other reason for a drive to just keep disappearing from the RAID like that?

I can still see it in /dev

ls /dev/sdd*

brw-rw---- 1 root disk 8, 48 Jan 18 22:20 /dev/sdd
brw-rw---- 1 root disk 8, 49 Jan 18 22:20 /dev/sdd1

I also noticed these messages:

[    5.779762] md: kicking non-fresh sdd1 from array!
[    5.906091] md: kicking non-fresh sdf1 from array!
[    5.918051] md: kicking non-fresh sdf2 from array!
[    5.920337] md: kicking non-fresh sdf3 from array!

What does "non-fresh" mean?

  • 4
    all your RAIDs are degraded and there's a lot going on in your dmesg as well – frostschutz Jan 19 at 19:32
  • 3
    Too much here. 1. Get a backup. 2. You can't see all of sdd*, for example you're missing sdd2. 3. "Non-fresh` means it's out of date with respect to its mirror, so it can't be added back – roaima Jan 19 at 21:20
  • I've fixed some of the issues by removing a couple disks and adding them back. But I'm still getting messages about "failed command: WRITE FPDMA QUEUED". Is it possible my SDDs (Samsung 860 QVO 1TB) just don't have the needed support for RAID? – Dave Jan 20 at 15:31
  • SDDs don't need "support for RAID". Try using the SDDs in a non-raid configuration and see if you still get errors; then you've ruled out for sure that RAID is involved somehow, and you need to figure out why your SDDs don't work with your other hardware. – dirkt Jan 26 at 18:04
  • You have SATA controller errors. First, check your cabling. If possible, replace cables for sdd and sdf (one disk and one SSD that have been kicked out). I also advise you to run a short test on all of your disks: smartctl -tshort /dev/sda. For instance for DISK in /dev/sd{a,b,c,d,e,f}; do sudo smartctl -tshort $DISK; done – wazoox Jan 30 at 16:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.