0

I just installed a new RAID 1 array as /dev/md2 to a server. It mounted correctly and I was able to read/write. I edited fstab and rebooted, but was unable to ssh in. I then plugged in a keyboard and monitor.

I received an error stating that a remote host that I normally connect to (a backup server that I mount to in fstab) couldn't mount. This was unsurprising as I had turned that server off while I was messing around so that there was no chance of me destroying the backups. I followed the advice of other posts and added a noauto line to fstab for this entry, and I was able to do this because I was able to access a shell after it spat out the error. I then used vim to edit fstab. Now that I think about it again, however, I find it distressing that this error didn't show up earlier, as I had rebooted once before I added the new array while the backup server was already off, and I didn't receive any errors then.

In any case, after adding this noauto option to the remote, upon reboot I still can't ssh in, and all that I'm getting from the monitor is

fsck from util-linux-ng 2.17.2
/dev/md1: clean, 3680768/60071936 files, 208210802/240284144 blocks

(/dev/md1 is mounted as /) There is a prompt, but it's not a shell. I can type anything, but pressing enter obviously doesn't do anything. When I use Ctrl+Alt+F2, I am taken to another empty prompt, but again there is no shell. This is also what I find in F3-F6.

This is what's really weird. When I then go back to the first prompt using Ctrl+Alt+F1, the screen is empty except for the cursor, which will now be at a random (as in it changes every time I reboot) vertical position.

Before I was getting something like [ 7.26####] hda-intel: no codec found!, which has something to do with the audio driver.

I need to get this server back up and running, but I can't edit fstab to remove the new array/undo my noauto edit, etc.

Any ideas?

Edit: I tried to enter recovery mode by holding down left shift during boot, and it allowed me to select a recovery mode (up to Ubuntu, with Linux 2.6.32-74-server), but this just runs a few things and then takes me straight back to the useless prompts above.

Edit2: I've just realized that I'm an idiot and the fsck from... is actually from the Alt+F7 console, and the randomly placed cursor/hda error is from the installation dialog (Alt+F1)

Edit3: After adding fastboot to try and skip over fsck, I receive the following:

[41.631382]EXT4-fs error (device md1): ext4_mb_generate_buddy: ECT4-fs:group 4033: 3895 blocks in bitmap, 3894 in gd
[41.632061] Aborting journal on device md1-8
[41.651633] EXT4-fs (md1): Remounting filesystem read-only
[41.652357] EXT4-fs(md1): delayed block allocation failed for inode 33031569 at logical offset 0 with max blocks 1 with error -30
[41.653074] This should not happen!! Data will be lost
[410653351] EXT4-fs (md1) ext4_da_writepages: jbd2_start: 1024 pages, ino 33031569 err -30

Edit4: I can now get into a bash terminal using grub init=/bin/bash, but the system is mounted as read only. I can confirm that the existing files are still in place, but I still can't boot. I have turned off my MD2 (new RAID 1), but this hasn't changed anything.

/proc/mdstat shows both RAIDs (md0, md1) as being active withboth drives up. Smart Control also verified that SDA and SDB (which both md1 and md0 are composed from) are working properly.

Now I'm stuck at fsck. It returns an error 3, meaning it found and corrected errors and that it needs to reboot. After that I get a mount error 4. The trouble is upon reboot, this continues ad infinitum, so whatever errors it thinks it is fixing aren't getting fixed.

Edit 5: I solved this last part through some combination of ext4 fsck, but unfortunately I can't describe what I did. That was a few hours ago and once I got that working I moved back to my original problem. If you're stuck on this part, sorry. I solved the rest of the problem, however.

Solution: I was able to get into a shell by doing the following: Upon reboot hold shift and press 'e' when selecting a kernel.

Modify the grub config that shows up by replacing quiet with init=/bin/bash to get to a terminal

This mounted in read only, so I ran mount -o remount,rw / to remount everything in read/write.

I then commented out my fstab changes and was able to boot again.

Unfortunately I now have another problem, but that's sufficiently different that I'll ask it in a different question.

  • Why don't you boot with a live CD/DVD and modify what ever you need from there? – YoMismo Jun 17 '15 at 15:44
  • The system may be simply running fsck (after an earlier reset or interrupted boot), which may take quite a while depending on your partition size, usage, fragmentation, etc. (maybe hours for terabyte size partitions!). If you see your harddisks being accessed I'd suggest giving it ample time to complete (leave it overnight?) – Dan Cornilescu Jun 17 '15 at 16:10
  • Currently burning a live CD, and none of the hard disks are being accessed. – barriboy Jun 17 '15 at 16:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.