4

I have a NAS running Ubuntu 16.04 LTS server i386 (kernel 4.4.211-0404211-generic). I use it with samba to share movies and music through my home network.

I've tried to copy ~ 100 Go of data from an HDD to a another recently bought, both are in ext4.

There are both mounted by fstab at startup like that :

UUID=... /media/HDD{number_of_hdd} ext4 defaults,errors=remount-ro 0 2

But everything I've tried has failed, in cause : everything is (partially) corrupted. All my files are readable, for example by VLC on my Windows PC thought my samba share, but there a lot of small pauses in the music or visual corruptions in videos, I have noticed that there a lot of decoded blocs marked as Dropped (discontinued) in VLC's statistics page, one more each time I noticed an audio or visual corruption.

So I've checked checksums (using 7z h {file} -scrcSHA256 and md5sum on both source and destination files), and every time they're different.

I've use cp and rsync to make the copy, both failed. Full commands I've used :

cp -r {source} {destination}
rsync -Pa {source} {destination}

I've checked smart values for both HDD, nothing is wrong, then I have done an fsck -f -y on both too, fsck tell me that everything is good. I have also made a memtest with memtest86+, and everything is good too. No updates available with apt update.

After hours of trying to found the problem, I've noticed things (both using cp or rsync) :

  • With small files (tested with 20Mo) created with truncate, the checksums are correct
  • With a folder which contain small files (around 5 files of 20Mo), the checksums are correct
  • If I try to copy the whole directory of 100Go, both commands tell me that everything is ok, but checksums don't match.

I have compared two files from the original directory and the corrupt one, and I have found some differences : 1 2]

I can send an archive with the original and the corrupt file if necessary.

Server specs :

==================================================
                            system     A7N8X-E
/0                          bus        A7N8X-E
/0/0                        memory     64KiB BIOS
/0/4                        processor  AMD Athlon(tm) XP 2800+
/0/4/9                      memory     128KiB L1 cache
/0/4/a                      memory     512KiB L2 cache
/0/26                       memory     3GiB System Memory
/0/26/0                     memory     1GiB DIMM DRAM Synchronous
/0/26/1                     memory     1GiB DIMM DRAM Synchronous
/0/26/2                     memory     1GiB DIMM DRAM Synchronous
/0/100                      bridge     nForce2 IGP2
/0/100/0.1                  memory     RAM memory
/0/100/0.2                  memory     RAM memory
/0/100/0.3                  memory     RAM memory
/0/100/0.4                  memory     RAM memory
/0/100/0.5                  memory     RAM memory
/0/100/1                    bridge     nForce2 ISA Bridge
/0/100/1.1                  bus        nForce2 SMBus (MCP)
/0/100/2                    bus        nForce2 USB Controller
/0/100/2/1      usb2        bus        OHCI PCI host controller
/0/100/2.1                  bus        nForce2 USB Controller
/0/100/2.1/1    usb3        bus        OHCI PCI host controller
/0/100/2.2                  bus        nForce2 USB Controller
/0/100/2.2/1    usb1        bus        EHCI Host Controller
/0/100/4        enp0s4      network    nForce2 Ethernet Controller
/0/100/8                    bridge     nForce2 External PCI Bridge
/0/100/8/4      enp1s4      network    88E8001 Gigabit Ethernet Controller
/0/100/8/a                  storage    SiI 3114 [SATALink/SATARaid] Serial ATA Controller
/0/100/9                    storage    nForce2 IDE
/0/100/1e                   bridge     nForce2 AGP
/0/1            scsi0       storage
/0/1/0.0.0      /dev/sda    disk       1TB ST1000DM010-2EP1
/0/1/0.0.0/1    /dev/sda1   volume     928GiB EXT4 volume
/0/1/0.0.0/2    /dev/sda2   volume     3070MiB Extended partition
/0/1/0.0.0/2/5  /dev/sda5   volume     3070MiB Linux swap / Solaris partition
/0/2            scsi1       storage
/0/2/0.0.0      /dev/sdb    disk       1TB ST1000LM048-2E71
/0/2/0.0.0/1    /dev/sdb1   volume     465GiB EXT4 volume
/0/2/0.0.0/2    /dev/sdb2   volume     465GiB Linux filesystem partition
/0/3            scsi2       storage
/0/3/0.0.0      /dev/sdc    disk       3TB ST3000DM007-1WY1
/0/3/0.0.0/1    /dev/sdc1   volume     2794GiB EXT4 volume
/0/5            scsi3       storage
/0/5/0.0.0      /dev/sdd    disk       1TB ST1000LM048-2E71
/0/5/0.0.0/1    /dev/sdd1   volume     931GiB Windows NTFS volume
/1              virbr0-nic  network    Ethernet interface

Hope this problem will be fixed quickly. Thanks

EDIT 1 (01/26/2020) : - I've run for 3h and half (2 pass precisely) the memtest86+, nothing wrong with the RAM.enter image description here - I've also checked dmesg for any message about corruption (CRC errors), nothing is reported, even if i make a copy (with also bad checksum at the destination) ...

- I am making 2 long S.M.A.R.T test on both HDD (the source and the destination) to check if nothing is wrong

- I have also noticed that if i stop the copy of files during the process (with CTRL + C for example), it seems to corrupt the ext4 filesystem, I don't know how and why ...

EDIT 2 (01/26/2020) : There's the two report for the both HDD in cause :

The Source HDD:

smartctl 6.5 2016-01-24 r4214 [i686-linux-4.4.211-0404211-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     ST3000DM007-1WY10G
Serial Number:    WFN2CMWR
LU WWN Device Id: 5 000c50 0cc67ff74
Firmware Version: 0001
User Capacity:    3 000 592 982 016 bytes [3,00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5425 rpm
Form Factor:      3.5 inches
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 1.5 Gb/s)
Local Time is:    Sun Jan 26 21:35:04 2020 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                    was never started.
                    Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                    without error or no self-test has ever 
                    been run.
Total time to complete Offline 
data collection:        (    0) seconds.
Offline data collection
capabilities:            (0x73) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    No Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                    General Purpose Logging supported.
Short self-test routine 
recommended polling time:    (   1) minutes.
Extended self-test routine
recommended polling time:    ( 359) minutes.
Conveyance self-test routine
recommended polling time:    (   2) minutes.
SCT capabilities:          (0x30a5) SCT Status supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   079   064   006    Pre-fail  Always       -       74326255
  3 Spin_Up_Time            0x0003   096   096   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       13
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   067   060   045    Pre-fail  Always       -       5351732
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       66 (132 227 0)
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       13
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   067   060   040    Old_age   Always       -       33 (Min/Max 30/38)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       5
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       23
194 Temperature_Celsius     0x0022   033   040   000    Old_age   Always       -       33 (0 23 0 0 0)
195 Hardware_ECC_Recovered  0x001a   079   064   000    Old_age   Always       -       74326255
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       55 (238 164 0)
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       2782921433
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       1050108775

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%        66         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

The destination HDD :

Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     ST1000LM048-2E7172
Serial Number:    ZDEBV755
LU WWN Device Id: 5 000c50 0b24d84fd
Firmware Version: SDM1
User Capacity:    1 000 204 886 016 bytes [1,00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Form Factor:      2.5 inches
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 1.5 Gb/s)
Local Time is:    Sun Jan 26 18:34:49 2020 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                    was never started.
                    Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                    without error or no self-test has ever 
                    been run.
Total time to complete Offline 
data collection:        (    0) seconds.
Offline data collection
capabilities:            (0x71) SMART execute Offline immediate.
                    No Auto Offline data collection support.
                    Suspend Offline collection upon new
                    command.
                    No Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                    General Purpose Logging supported.
Short self-test routine 
recommended polling time:    (   1) minutes.
Extended self-test routine
recommended polling time:    ( 162) minutes.
Conveyance self-test routine
recommended polling time:    (   2) minutes.
SCT capabilities:          (0x3035) SCT Status supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   083   064   006    Pre-fail  Always       -       193912808
  3 Spin_Up_Time            0x0003   099   099   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       228
  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   070   060   045    Pre-fail  Always       -       9188472
  9 Power_On_Hours          0x0032   092   092   000    Old_age   Always       -       7832 (182 235 0)
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       197
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   074   058   040    Old_age   Always       -       26 (Min/Max 26/31)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       78
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       1655
194 Temperature_Celsius     0x0022   026   042   000    Old_age   Always       -       26 (0 16 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       1
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       7545 (117 142 0)
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       5699868670
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       10646554103
254 Free_Fall_Sensor        0x0032   100   100   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%      7832         -
# 2  Short offline       Completed without error       00%      7830         -
# 3  Short offline       Completed without error       00%      3141         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

EDIT 3 (02/15/2020) : I finally fix the problem by setting libata.force=noncq linux paramater, it seems that is clearly a bug from the kernel and it will should be definitely fixed soon (https://bugs.launchpad.net/ubuntu/+bug/1861300)

  • 3
    If you run md5sum a bunch of times times on a large file (at least 2x size of RAM), do you keep getting the same answer? Or is it sometimes different? If so, does that happen on both drives, or just one? – derobert Jan 25 at 19:25
3

(Note: This presumes you're not getting a bunch of kernel errors (check dmesg or journalctl -b -k) or tons of CRC errors indicated in drive SMART status. If you are... there are a few software things to try first, like turning of NCQ.)

Usually, this means bad RAM. Even when memtest86+ passes (how long did you run it for?) Unless you have ECC RAM, which I doubt from those specs.

Make sure you haven't done something crazy, like find 1+ meter SATA cables and wrap them around the CPU. Although SATA data transfer has CRCs, so you should be getting tons of errors if you're getting corruption here. SATA cables are cheap, you can always try replacing them.

The next step, if you don't just want to replace the RAM, is to try to narrow down when the corruption is happening.

On each drive, repeatedly run md5sum or similar on a large file showing the issue (needs to be something like 2x RAM, to stop it from being checked from cache) or set of files. Do it a lot of times, like for hours. Do you always get the same result? If not, then there is corruption on the read path; if you always get the same result, then there probably isn't corruption on read. That'd make RAM unlikely.

If you get read corruption on both disks, start with replacing the RAM. If that doesn't fix it, you can try power supply and finally SATA controller (which is likely soldered to the mobo, so you'd have to replace that).

If you get read corruption on one disk (not both), replace the disk. If that doesn't fix it, and you have a backplane (for hot swap in the server), it may be defective. You can try replacing the cables as well. Try a different SATA port. The presumption here is that one bad disk may happen, but two is pretty unlikely. Honestly... I'd swap RAM before presuming two bad disks.

If both disks consistently read back the same data, first confirm you're actually checking enough data to be sure its not cached; I'd want at least twice RAM. You'd then repeatedly write some known data to each disk, and see if reading it back gives a different value. Then pretty much the same solutions as above.

PS: Corruption like this is insidious. In particular, it may have corrupted random bits of your Linux distro, not just your data. After fixing the cause, it's usually best to re-install. At minimum, you need to check every distro-provided file against known-good checksums; some distros provide utilities for doing that. That still won't confirm no damage to dynamic distro data files (e.g., installed package lists), but at least you can be sure the binaries are OK.

  • 1
    I've run memtest86+ about 1 hour (time takes to make 1 pass, so all tests had been passed). All my SATA cables are less than 30cm. And the SATA PCI card is at the last bottom PCI slot (that allow to use SATA on this motherboard, cause the integrated SATA controller doesn't allow disk above 1TB), far from the CPU. It seems that i never got read corruption, but it seems to be only inter-disk write corruption. I meens if I wrote data with Samba, i never had any corruption on those disks ! – Xenoxis Jan 26 at 0:49
1

Once, long time ago I had same issue. Problem was in broken BIOS. It is unlikely that this is RAM, as if that was RAM you should also get random crashes 'for free' and problem will happen on both drives, not on one (Am I corretly assuming that problem only happens on new drive?)

I would focus on this: Exclude copy and from process. Replace it with write only. Create files with dd with large blocksize (dd if=/dev/zero of=myfile bs=1M count=100). Find exact size then it breaks.

  • The server has runned for 4 month without any interruption, so if the RAM was wrong, the system should have been crash without any reason. – Xenoxis Jan 26 at 0:32
  • And no, the corruption his on the both drives. Indeed, i have copied 1TB of data from a drive (that I call HDD1) with NTFS to the new one (3TB, I call it HDD3) that I have formatted on ext4, but all of the data was corrupted (I've used rsync). Then i've decided to put ext4 on the HDD1, and I've copied a partial backup on HDD3, about ~100 Go of the 1To of loss data (backup that I have copied from Windows with Samba, and everything was good ! No error in checksum, so no corruption !). Then, i've decided to move the backup from HDD3 to HDD1 (with rsync) but once again, the data was corrupted ! – Xenoxis Jan 26 at 0:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.