3

I'm working with Amazon Linux in AWS and attempting to setup multiple file systems on a single EBS Volume (block device) for compliance reasons. I'm doing this by mounting an EBS Volume to an already-running EC2 Instance, executing a series of commands, unmounting it, creating a snapshot and then turning it into an AMI.

I can get everything working with a "basic" set of commands that simply delete and re-add an existing partition. But when I add a second partition, I'm no longer able to launch any EC2 Instances from that AMI. Instead, using the "Get Instance Screenshot" functionality in AWS, the EC2 Instance never boots and I see this:

enter image description here

When I execute the following commands from a host EC2 Instance on which I've mounted an EBS Volume, everything works as expected:

# Make a tar of the current EBS Volume
umount -l /mnt/ebs-volume
mount -o ro /dev/xvdf1 /mnt/ebs-volume
tar -cf /tmp/ebs.tar --exclude='./dev/*' --exclude='./proc/*' --exclude='./sys/*' /mnt/ebs-volume
umount /mnt/ebs-volume

# Replace the old partition with one new partition and format it as ext4
sgdisk --delete 1 /dev/xvdf
sgdisk --new 1:0:+7800M /dev/xvdf && sgdisk --change-name 1:Linux
/sbin/mkfs.ext4 -F -m0 -O ^64bit /dev/xvdf1 && e2label /dev/xvdf1 /

# Mount the new partition and restore the snapshot
mount /dev/xvdf1 /mnt/ebs-volume
cd / && tar -xf /tmp/ebs.tar --acls --selinux --xattrs && rm /tmp/ebs.tar

# I don't make any updates to the /etc/fstab file

But when I execute these commands instead, the EC2 Instance fails to boot as shown above:

# Make a tar of the current EBS Volume
umount -l /mnt/ebs-volume
mount -o ro /dev/xvdf1 /mnt/ebs-volume
tar -cf /tmp/ebs.tar --exclude='./dev/*' --exclude='./proc/*' --exclude='./sys/*' /mnt/ebs-volume
umount /mnt/ebs-volume

# Replace the old partition with two new partitions and format them as ext4
sgdisk --delete 1 /dev/xvdf
sgdisk --new 1:0:+7800M /dev/xvdf && sgdisk --change-name 1:Linux
sgdisk --new 2:0:+100M /dev/xvdf
/sbin/mkfs.ext4 -F -m0 -O ^64bit /dev/xvdf1 && e2label /dev/xvdf1 /
/sbin/mkfs.ext4 -F -m0 -O ^64bit /dev/xvdf2

# Mount the new partitions and restore the snapshot
mount /dev/xvdf1 /mnt/ebs-volume
mkdir -p /mnt/ebs-volume/boot && mount /dev/xvdf2 /mnt/ebs-volume/boot
cd / && tar -xf /tmp/ebs.tar --acls --selinux --xattrs && rm /tmp/ebs.tar

# I now execute commands that write the below /etc/fstab file to the volume at /mnt/ebs-volume/etc/fstab

/etc/fstab:

LABEL=/     /               ext4    defaults,noatime    1   1
/dev/xvda2  /boot           ext4    defaults,noatime    0   0
tmpfs       /dev/shm        tmpfs   defaults            0   0
devpts      /dev/pts        devpts  gid=5,mode=620      0   0
sysfs       /sys            sysfs   defaults            0   0
proc        /proc           proc    defaults            0   0

Why would adding a separate file system for /boot cause this error? Am I missing something in my /etc/fstab or perhaps another file somewhere since I suppose this new configuration is now no longer the same as the original configuration.

  • 1
    By putting /boot, and correspondingly the kernel image, on a different partition, when the system tries to boot, it cannot find the kernel image. Are you using grub for boot loading? – Andy Dalton Feb 15 '17 at 3:36
3

This may require some pre-history of the PC boot sequence to fully explain. (Modern PC's with UEFI do things differently, but the logic is similar).

So let's look back at the 80s, when PCs first started to have hard disks.

The BIOS would load the first sector of the hard disk. This contained the Master Boot Record (MBR), which was a combination of code and the partition table. There was only 512 bytes to hold all of this, so coding was tight.

The MBR would look at the partition table and find out which partition was active and where it started. It would then load a secondary boot system, which was stored inside that partition. (Historically this meant the secondary boot loader had to be within the first 504Mb of disk). This code knew about the filesystem and could load the OS (typically IO.SYS, MSDOS.SYS, COMMAND.COM). And thus DOS booted. A typically new PC would require "fdisk /mbr" to install this primary boot sector.

The fact that it was software in the MBR made the boot process flexible and allowed alternate boot loaders. An early boot loader for Linux was LILO ("Linux Loader"). This had a primary loader, a secondary loader. It knew about standard Linux filesystems and was able to dual boot Linux and DOS (and Windows).

Later GRUB (and then GRUB2) came along. But they all follow the primary/secondary boot loader process.

Now what you're doing is moving the secondary boot process when you modify the /boot partition. The (pretty dumb) primary boot loader doesn't know where the find the smart part. And so your VM fails to boot.

What you need to do is the modern equivalent of the old "fdisk /mbr" process; you need to tell your MBR where to find the secondary loader.

How you do this depends on what boot loader you're using. It may be "grub-install" or "grub2-install" or "lilo". It'll depend on the OS variant (CentOS, Ubuntu, Debian, Amazon... they may all be different).

This doesn't tell you what to do to fix your build, but at least, now, you should understand why your OS fails to boot!

  • Thanks for this detailed and very helpful response! I'm going to try out a few things and then will report back with the official item I needed to update. I'll plan to mark this response as the accepted answer. – Josh Padnick Feb 15 '17 at 21:53
  • Update. I was ultimately unable to update the MBR in Amazon Linux using the pre-installed tooling. grub-install resulted in a "does not have any corresponding BIOS" error. grub-mkconfig was not present. This meant I had to stick with the existing MBR configuration, so the way I solved this was to keep /boot and / on the same partition, and to add separate partitions beyond that. Because your comment answered exactly the right issue, I'll mark it accepted now. – Josh Padnick Feb 17 '17 at 3:13
0

Note that the boot volume in AWS is kind of special, I haven't been able to find a place to confirm it and explain why, but the root volume is attached as sda1 (xvda1), note the partition number 1 at the end, see http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/RootDeviceStorage.html. I think this might be the reason the bios fails to boot from a partition that itself contains a partition table and more partitions.

I don't think you can avoid using more than one volume if you need more than one partition (without using ugly tricks like loop devices attached to big files in the root partition :D).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.