After rebooting my ZFS pool was unable to open my main pool. The exact error that I'm getting is: "The pool metadata is corrupted and the pool cannot be opened"
When I checked the zpool configuration using zpool status (from the recovery console), the configuration that it displayed was all wrong. The configuration listed several drives that I had just moved to other drives.
Currently the output of zpool status looks like this:
pool: pool state: FAULTED status: The pool metadata is corrupted and the pool cannot be opened. action: Destroy and re-create the pool from a backup source. see: http://zfsonlinux.org/msg/ZFS-8000-72 scan: resilvered 511G in 12h39m with 0 erors on Sat Mar 14 06:14:34 2015 config: NAME STATE READ WRITE CKSUM pool FAULTED 0 0 1 corrupted data raidz1-0 ONLINE 0 0 8 wwn-0x50014ee05943ce36-part4 ONLINE 0 0 0 wwn-0x50014ee05943ce36-part5 ONLINE 0 0 1 wwn-0x50014ee05943ce36-part6 ONLINE 0 0 0 wwn-0x50014ee05943ce36-part7 ONLINE 0 0 0 wwn-0x50014ee05943ce36-part8 ONLINE 0 0 1
I'm running Fedora 20, kernel 3.18.7-100
I checked with zdb by running
zdb -l /dev/disk/by-id/wwn-0x50014ee05943ce36-part4 and it shows 4 labels, all displaying the correct configuration which it was before I rebooted.
Is it possible that ZFS scans my disks, finds one of the old partitions with the old configuration and uses that instead of the new one? (zdb -l on the other partitions all show different configurations than the one I'm getting when I booting, except part8, which shows that it is replacing part8 by another disk) How can I tell ZFS to use the correct configuration so that I can import and open my pool?
I've got everything running again but I unfortunately don't really know why things are working again. What I did was booting into the live CD of Fedora. I installed all the necessary tools to use ZFS and from there I tried to look at the status the zpools were in. There it gave me a different configuration (the right one). Why that was different from the one I got when I booted and ended up in recovery mode, I don't know. But after I tried to import it, it said one or more devices was still in use. I finally found that the partitions that were used by ZFS, which were previously used by mdadm as RAID devices, were still being detected as being part of the RAID array. I removed the RAID array, zeroed out the superblocks. Once I did that, I was able to import the pool. I scrubbed the entire array and no errors were reported. I rebooted into the actual system, and everything worked as it should. Maybe the scrub fixed some problems, maybe the old RAID-array got in the way of ZFS. (I read somewhere that mdadm starts before ZFS)
After doing a kernel upgrade, I once again was stuck at the recovery mode, because it couldn't open my ZFS pool. I went back to my live-cd and recreated dracut, using
dracut -f "initramfs-$(uname -r).img" $(uname -r) (making sure that the kernel matched the kernel that I was actually using in my system and not within the live-system) This got me up and running again, so I guess it is required to recreate dracut once you change your zpool-root configuration.