1

I had to open my file-server's housing today to replace a faulty fan. What I didn't see was that one of the sata-cables was not properly connected. The 1st thing I did after a reboot was a check of the RAID status and it showed immediately that one drive is missing. Till this moment the device was not used (however it was mounted, so I'm not 100% sure that system did nothing). I stopped md0 and re-plugged in the cable:

mdadm --stop /dev/md0
poweroff

After another reboot I checked the removed drive:

 mdadm --examine /dev/sdd1
 ...
       Checksum : 3276bc1d - correct
         Events : 315782

         Layout : left-symmetric
     Chunk Size : 32K

      Number   Major   Minor   RaidDevice State
this     0       8       49        0      active sync   /dev/sdd1

   0     0       8       49        0      active sync   /dev/sdd1
   1     1       8       65        1      active sync   /dev/sde1
   2     2       8       33        2      active sync   /dev/sdc1
   3     3       8       17        3      active sync   /dev/sdb1

I was a bit surprised that it was shown as active (even if earlier mdadm said that this device was removed from array) and the checksum was OK. I recreated RAID with:

mdadm --assemble /dev/md0 --scan

The command mdadm --detail /dev/md0 showed that all drives were running and system was in "clean" state. I mounted the device md0 and then came hic-cup. I wanted to work on one of the last files that I had been using before all the situation and it was not there. In another place I missed actually all files from the directory where I was working. As far as I can see most of the files that are older than a few days are intact but some newer ones are missing.

Now the big question: what would be your advice? Is there a way to get this data? I thought about removing the drive that was earlier labeled by mdadm and rebuild array with another HDD.

UPDATE

I started to back-up the drives today. After mounting md0 as read-only I run rsync to another server. Now curious thing. I moved a week ago some directories to other array. rsyns has shown following info on these removed dirs:

file has vanished: "/MD0/Data/_NMR_"
  • A failing drive should not cause files going missing, so something else must have happened. Rebuilding won't help, esp. not after mounting the filesystem in read-write mode. If you have any logs you should check them for mdadm/filesystem messages. – frostschutz Jun 24 '13 at 10:20
  • I've found that after re-assemble the "broken" drive is on another place in RAID (before it was listed on 3rd place, then on 4th). Can this have influence on rebuilt process? If yes, how to reassemble the array properly? I'm sure the SATA-cables are connected still in the same order to the controller. – Kris_R Jun 26 '13 at 13:46
  • If the drives were in a wrong order, it's highly unlikely you'd be able to mount at all... esp. with smallish chunksize as is the case in your setup. Also, all files would be corrupt, not just a few missing... – frostschutz Jun 26 '13 at 15:28
0

Maybe it will help somebody.

I didn't write it before but all four partitions had the same count of events

mdadm --examine /dev/sd[a-z]1 | egrep 'Event|/dev/sd`'
mdadm: No md superblock detected on /dev/sda1.
         Events : 315786
         Events : 315786
         Events : 315784
         Events : 315786

Still, after some reading I decided to remove the "failed" drive and re-assamble my md0 device.

mdadm --manage --set-faulty /dev/md0 /dev/sdd1
mdadm /dev/md1 --stop
mdadm --assemble /dev/md0 /dev/sd[bce]1 --force 
mdadm --manage /dev/md0 --add /dev/sdd1

Please, don't ask me why it worked. The important part for me is that I got back all the files (file allocation table shows proper content of directories. All missing files are there.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.