2

Background

My objective is to use Packer to create an Amazon Machine Image (AMI) with several different paths mounted to different filesystems to improve security. E.g. /tmp should be mounted to a filesystem with the noexec option.

The fact that I want to create an automated process for making an AMI means that I can't execute re-mounting commands in the instance itself, so I am instead using the Packer amazon-chroot builder. This means I run an EC2 Instance, and run Packer from that EC2 Instance. Packer will then mount an EBS Volume taken from an EBS Snapshot used with a "source AMI". I now need to perform some operations on the mounted EBS Volume.

I am taking inspiration from a recent presentation on this topic whose slides are at http://wernerb.github.io/hashiconf-hardening.

My Question

When my EBS Volume (Block Device) is first mounted, here are the partitions I see on it from gdisk -l /dev/xvdf:

Disk /dev/xvdf: 16777216 sectors, 8.0 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): 726A877B-31D7-4C00-99E4-5A2CCB8E0EAD
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 16777182
Partitions will be aligned on 2048-sector boundaries
Total free space is 2014 sectors (1007.0 KiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            4096        16777182   8.0 GiB     8300  Linux
 128            2048            4095   1024.0 KiB  EF02  BIOS Boot Partition

I then perform the following operations:

  • Delete the "Linux" partition with sgdisk --delete 1 /dev/xvdf
  • Create an LVM Volume Group with lvm vgcreate -y main /dev/xvdf1
  • Create a series of LVM Logical Volumes and format them each with a command like /sbin/mkfs.ext4 -m0 -O ^64bit "/dev/main/lvroot"
  • Mount them all and copy a bunch of files over
  • Update /etc/fstab on the attached EBS Volume as follows (this is /mnt/ebs-volume/etc/fstab from the perspective of my host system):

/etc/fstab I write to /dev/xvdf1:

#
/dev/mapper/main-lvroot         /               ext4    defaults,noatime    1   0
tmpfs                           /dev/shm        tmpfs   defaults            0   0
devpts                          /dev/pts        devpts  gid=5,mode=620      0   0
sysfs                           /sys            sysfs   defaults            0   0
proc                            /proc           proc    defaults            0   0
/dev/mapper/main-lvvar          /var            ext4    defaults            0   0
/dev/mapper/main-lvvarlog       /var/log        ext4    defaults            0   0
/dev/mapper/main-lvvarlog/audit /var/log/audit  ext4    defaults            0   0
/dev/mapper/main-lvhome         /home           ext4    defaults            0   0
/dev/mapper/main-lvtmp          /tmp            ext4    defaults            0   0

Finally, Packer unmounts /dev/xvdf and makes an Amazon Machine Image (AMI) based on the contents of that EBS Volume.

So far so good, except that when I go to launch a new AMI, it doesn't actually boot. I can't connect via SSH, and "View System Logs" via AWS shows nothing. So I'm assuming I'm messing something up around that "128" partition that contains the "BIOS Boot Partition". I'm also confused about how my LVM-created Logical Volumes are supposed to become "activated" when the new EC2 Instance boots up.

Basically, I'm missing a mental model for what needs to exist in that Boot partition and how the EC2 Instance can boot and run LVM if I've used LVM to create the root volume itself? I'm wondering if I need to create a special partition on /boot, but what do I put in that? Should I in fact have three partitions on my /dev/xvdf, the "BIOS Boot Partition", a "traditional" (ext4-formatted) partition for /boot and an LVM-managed partition for everything else?

  • The key to booting off an LVM logical volume is to have LVM support in the kernel (of course) and to also boot with an initrd which contains LVM support. Creating an initrd is non-trivial, so I suggest you look into whether your Linux distro is set up to boot from LVM. And also make sure EC2 is booting the kernel with the initrd. – Emmanuel Rosa Feb 14 '17 at 15:16
  • Thanks for this clarification. LVM support in the kernel isn't an issue for the latest version of Amazon Linux, but it appears that Amazon Linux is still using GRUB 0.97, which doesn't explicitly support LVM, so that was the issue. Thanks for your help! If you want to add this as a response, I'll mark it as the correct response. – Josh Padnick Feb 14 '17 at 19:40
  • Ahhh, yes. You can still put your rootfs on an LVM logical volume. But due to using GRUB "legacy" you'll need a /boot partition --not LVM volume-- with a supported filesystem (such as ext4) so GRUB can load your kernel and initrd. – Emmanuel Rosa Feb 14 '17 at 22:08
  • Actually, I did create a /boot partition with a kernel and initrd file, but I couldn't figure out how to get my root fs loaded after that? I tried referencing LVM devices like /dev/main/lvroot in my /etc/fstab but the system still wouldn't even boot. My plan is to move off of LVM, but any other thoughts for solving this? – Josh Padnick Feb 14 '17 at 22:23
  • I'm guessing my initrd doesn't support LVM either, so direct partitions is probably the way to go. – Josh Padnick Feb 14 '17 at 22:31
2

The issue turned out to be unrelated to LVM. As clarified in Why does this change render my Block Device unbootable?, the real issue is that by separating / and /boot into two separate partitions, the MBR configuration was no longer correct. I was unable to update GRUB configuration files to fix this, so ultimately I had to keep / and /boot on the same partition and add my other partitions separately. Not ideal but it works.

1

The key to booting off an LVM logical volume is to have LVM support in the kernel (of course) and to also boot with an initrd which contains LVM support. Creating an initrd is non-trivial, so I suggest you look into whether your Linux distro is set up to boot from LVM. And also make sure EC2 is booting the kernel with the initrd.

  • My issue actually turned out not to relate directly to LVM compatibility but to the way I'm creating my partitions using sgdisk. It seems that when I create the exact partition I just deleted, it works fine, but when I add a second partition, it stops working. Therefore, I'll upvote this but won't mark it as the approved answer. Many thx for your help! – Josh Padnick Feb 15 '17 at 2:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.