2

I have a problem with a filesystem here on a storage machine. We noticed, that many of the data that comes out of the systems seams to be corrupt, but only with minor problems like CRC errors with self-verifing installers or small picture errors in movies. While tracking down the problem, i endet up in a test with 3 files with about 900MB each. The ext4 filesystem is mounted read-only, but every time i do a md5sum on the files, the result differs:

$  ls -l
-rw-rw-r-- 1 samba samba 922789695 Jan  7 21:47 File1
-rw-rw-r-- 1 samba samba 939080225 Jan  7 21:54 File2
-rw-rw-r-- 1 samba samba 996515494 Jan 14 21:13 File3

$  md5sum *
9449c8e4fd2869a7969017db266451b0  File1
016b5c2e8b535ec922f5efb4ec9082bc  File2
5576aeb34575e07171fa835a79fec147  File3
$  echo 3 > /proc/sys/vm/drop_caches # (clear file cache of the kernel)
$  md5sum *
3f03edec64e22de384fd3d2cff0e3730  File1
32b53ee1dd3f5c9796322cabe4f8c0da  File2
35af5c433d0725ab0892d4517faeceea  File3
$  echo 3 > /proc/sys/vm/drop_caches
$  md5sum *
593d83e084387a8d5bd9b445032a5669  File1
4f8b76249b96a1a29bdd748167c41bda  File2
8b5bab8a153eb6e33dc3cd7d23362090  File3
$  echo 3 > /proc/sys/vm/drop_caches
$  md5sum *
d716d9c4acbd3ade450bab46903810d9  File1
68ede84d1396075ffe8a9228966cc148  File2
b8d75123b2d5b18c0d2827a448f53086  File3
$  echo 3 > /proc/sys/vm/drop_caches
$  md5sum *
c991bcca3bc2f39fdd143f8460935646  File1
73e6301b28c3b1b0bb95df52ea5794dd  File2
a202e88343d6e7bc4dce808b885ad013  File3

First I let e2fsck check the whole disk. It found a few problems, but it finds other errors on every new run. I think it got other reads each time same as md5sum and the problem is on another layer. The whole thing is inside a xen vm, but i don't think that detail matters.

The architecture is like:

ext4
 |
dm-crypt
 | (xen blk between here)
md-raid5 (softraid)
 |
 +---+-----------------------------+
     |                             |
mainboard sata         +---------pcie---------+
     |                 |                      |
  3 disks         sata controller(jbod)   sata controller(jbod)
 (1 failed)            |                      |
                    2 disks                2 disks

lspci output of the sata controllers:

00:12.0 SATA controller: Advanced Micro Devices [AMD] nee ATI SB600 Non-Raid-5 SATA
02:00.0 RAID bus controller: Silicon Image, Inc. SiI 3132 Serial ATA Raid II Controller (rev 01)
03:00.0 SATA controller: JMicron Technology Corp. JMB363 SATA/IDE Controller (rev 03)

While i was searching for the problem, one of the 7 disks failed and the raid is currently running with only 6 disks until the replacement arrives. Maybe this can be part of the problem? It existed definitly before the failure, but now the raid should be in a vulnerable but stable state...?

Whats going on here?

Improve this question
8
  • 1
    So this is a pure md raid, no BIOS fakeraid or a hardware RAID controller between the disks and the system? The disks are directly connected via SATA to the onboard SATA controller of the mainboard? I'm suspecting some kind of controller issue here.
    – Martin von Wittich
    Jan 24, 2014 at 21:44
  • Yes, a flakey disk can cause you to see inconsistent data. In my experience, reading from failing disks will sometimes return bad data prior to the disk giving notification that it has hard errors, despite us wishing that it were the other way around. If you have time, run a self-test on the disk (smartctl can do this) to see whether the disk is actually bad. Other hardware can cause errors, too: controllers, riser cards, port multipliers, cables.
    – Mark Plotnick
    Jan 24, 2014 at 22:02
  • Yes, it is a pure md raid. In fact 3 discs are direcly on the mainboard and the other 4 are connected with two pcie fakeraid adapters à 2 ports in jbod mode. I updated the diagram and added a lspci output. Can I test the controllers directly in any way? My idea is to detach everything, plug an empty disk to it and do testing on that. But maybe there is an easyer way?
    – a4c8b
    Jan 24, 2014 at 22:02
  • 1
    I'd check the memory
    – Stéphane Chazelas
    Jan 24, 2014 at 22:43
  • YESS!! Thank you, I think I found the problem: After a while of plugging arround different setups I replaced the SiI controller(the cheap one, hehe) with an old PCI one and the problem seems to be solved. Is it dangerous to try to re-attach the failed disk? It has no smart errors, can be formatted/used as normal in my test - I think the failure is related to the faulty sata controller.
    – a4c8b
    Jan 24, 2014 at 23:10

1 Answer 1

Reset to default
0

I think I found the problem: After a while of plugging arround different setups I replaced the SiI controller with an old PCI one and the problem seems to be solved.

Improve this answer

Not the answer you're looking for? Browse other questions tagged .