Sizing a disk image/filesystem to hold exactly a certain set of files with near-zero free space

Question

I need to construct, in code suitable for scripting/make/automation, a disk image (two partitions, a FAT filesystem and an ext4 filesystem, for UEFI/Syslinux USB Linux booting [OS installer], which will never change in production) of a size that holds exactly (or very close to) a set of files of arbitrary size. In other words given a set of built files, I need to know how to generate FAT and ext4 filesystem images, and the disk image that is partitioned to hold them, that have sizes computed to result in as close to zero space free as reasonable to have. It's OK to err on the side of a little extra room but it's not OK to fail when someone adds N+1 bytes to the file two years from now. This needs to be suitable for a makefile, i.e., trial and error doesn't cut it (although I suppose if worse came to worst, an iterative solution with a threshold might work). This is something like an ISO-9660 image (which it used to be on a previous project, but Syslinux doesn't support ISO-9660 images on UEFI).

I'm creating the files using dd (to allocate the disk image), parted, dd (to allocation the FAT filesystem), mkfs.vfat, dd (for the ext4), mkfs.ext4, kpartx (to map the partitions), dd (to write the FAT partition), dd (to write the ext4 partition), and finally dd to write the disk image to a USB for booting on actual hardware.

My current idea is to use du to determine how much space the files take on the build disk, then add some margin for additional filesystem and partition overhead, and margin of error. So I need to know the block counts for each of the dd's given the du outputs.

Another option is to construct a fixed-size large image, write the files, then resize the FAT, ext4, and partitions to minimum size. ext4 filesystems can be shrunk and I see FAT filesystems can be shrunk. But then you're still left with the problem of computing how much to shrink it to. Wondering if anyone has done this before had has some concrete ideas (or sample code).

Answer 1

I ended up using a combination of fudge factors and an iterative approach (as a fail-safe). This doesn't need to be as complicated as I originally envisioned. Turns out, for now, little fudging was required for FAT, but a huge amount of negative fudging was required for ext; with a zero fudge factor for ext4, I had plenty of free space (over 21M). I converted to ext2 (who needs a stinkin' journal?!), increased my block size, and carefully counted the inodes I needed, and got even more megabytes of free space out of it. I suppose I could have gotten the "real size" from du and worked up from there, but I figured counting the overhead, even if they are different filesystems, would be a closer approximation.

# Estimated filesystem overhead, in 512-byte blocks
FS_ESP_FUDGE=256 
FS_ISO_FUDGE=-80000 # Wow!
FS_FUDGE_INCR=1024
...
read ESP_RSIZE d < <(du --summarize --block-size=512 $ESP)
read ISO_RSIZE d < <(du --summarize --block-size=512 $ISO)

success=false
until $success; do 
    let ESP_SIZE=ESP_RSIZE+FS_ESP_FUDGE
    let ISO_SIZE=ISO_RSIZE+FS_ISO_FUDGE
    let ESP_START=2048
    let IMG_SIZE=ESP_SIZE+ISO_SIZE+ESP_START
    let ESP_END=ESP_START+ESP_SIZE-1
    let ISO_START=ESP_END+1

    success=true
...
    sudo /sbin/mkfs.vfat /dev/mapper/$p1 -F 16 \
        || error_exit "mkfs.vfat failed" 5
    # -N: Count the inodes (all files, plus . and .. for each directory,
    # which I can't get "find" to include).  
    sudo /sbin/mke2fs -b 4096 -N $(( $(find $ISO | wc -l ) + 2 * $(find $ISO -type d | wc -l))) -m 0 -q /dev/mapper/$p2 \
        || error_exit "mke2fs failed" 6
...
    if ! tar -C $ESP -c --exclude-vcs --exclude-backups . | \
        sudo tar -C mnt/esp -x; then
        {
            read
            read fs onek used avail use rest
        } < <(df mnt/esp)
        # Are we out of disk space? If not, bail, else increase margin, retry
        [[ $onek -ne $used || $avail -ne 0 || $use != "100%" ]] && \
            error_exit "esp tar failed" 9
        let FS_ESP_FUDGE=FS_ESP_FUDGE+FS_FUDGE_INCR
        success=false
    fi
    if ! tar -C $ISO -c --exclude-vcs --exclude-backups . | \
        sudo tar -C mnt/iso --owner=root --group=root -x ; then
        {
            read
            read fs onek used avail use rest
        } < <(df mnt/iso)
        # Are we out of disk space? If not, bail, else increase margin, retry
        [[ $onek -ne $used || $avail -ne 0 || $use != "100%" ]] && \
            error_exit "iso tar failed" 10
        let FS_ISO_FUDGE=FS_ISO_FUDGE+FS_FUDGE_INCR
        success=false
    fi
    $success || echo "Whoops, I guessed too small; please adjust fudge factor.  Retrying ..."
...
done