How to replace linux distro ISO on bootable USB keeping extra data partition intact and accessible?

Question

USB sticks are getting larger. I'd like to store data in addition to having USB bootable. And I know how: 1. write ISO to USB stick 2. Add partition via Gnome-disks (both GUI).

Now I want to be able to replace ISO keeping that extra data intact and accessible. How to do that? Preferably w/out manual editing hex data on the disk, but this solution is better than no solution. TIA

I've tried and failed as below. I've read How to copy the partition layout of a whole disk using standard tools where using sfdisk was advised.

Most ISO's I've encountered look as one below:

sfdisk --dump linuxmint-20.2-cinnamon-64bit.iso 
label: dos
label-id: 
device: linuxmint-20.2-cinnamon-64bit.iso
unit: sectors
sector-size: 512

linuxmint-20.2-cinnamon-64bit.iso1 : start=           0, size=     4222944, type=0, bootable
linuxmint-20.2-cinnamon-64bit.iso2 : start=         640, size=        7936, type=ef

That is some partition table sfdisk cannot write back (start 0 is not OK for it).

But after creating a partition via gnome-disks:

sudo sfdisk --dump /dev/sda
label: dos
label-id: 
device: /dev/sda
unit: sectors

/dev/sda1 : start=     8433664, size=    57104384, type=83
/dev/sda2 : start=        4800, size=        8496, type=ef
# (note: the ISO I've tried to change from had start=4800 for ef).

So I've saved that new line /dev/sda1 to say a.part, wrote new ISO to USB, created partition via gnome-disks, confirmed via sfdisk --dump, then deleted data partition via gnome-disks and did sfdisk --append /dev/sda < a.part (see part of output below). I saw "The partition table has been altered.", my data partition in new partition table. But USB stick no longer booted.

/dev/sda3: Created a new partition 3 of type 'Linux' and of size 24.4 GiB.
Partition #3 contains a btrfs signature.
/dev/sda4: Done.

New situation:
Disklabel type: dos
Disk identifier: 

Device     Boot    Start      End  Sectors  Size Id Type
/dev/sda2           4800    13295     8496  4.2M ef EFI (FAT-12/16/32)
/dev/sda3       14368768 65538047 51169280 24.4G 83 Linux

The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

Added (about Ventoy):

I'm using Ventoy already. I still want to have some USB with my favorite system so that I insert the stick, press power and the system loads unattended while I do other stuff. Also I recall Vetroy GUI glitched on some old computer, so again dedicated directly bootable stick is IMO useful.

However these issues are mitigated if Ventoy has option to load default specified file (with delay like GRUB boot menu) - could not find such googling and reading https://www.ventoy.net/en/faq.html. As seems Ventoy internally uses GRUB boot I opened config file to see if I can add default boot option.

Answer 1

Good things first: since modern computers for > 10 years have supported UEFI boot from GPT-partitioned USB devices, there's absolutely nothing special about the formatting/partitioning of your stick that you need to make it bootable!

All that is needed is that there is a FAT32- or exFAT-formatted partition that contains an EFI/boot/boot{architecturename}.efi boot loader file; I'll assume this is about x86_64, so that'd be EFI/boot/bootx64.efi. That bootloader can do whatever it pleases – from directly loading a Linux kernel that proceeds to boot a single installation from the USB stick, to loading a GRUB that then loads one of many Linuxes (or other OSes), to chain-loading the bootloader from a CDROM disk image (which is what .iso files actually are – it's a historical curiosity these are still the dominant way of getting installers onto USB sticks); technically, it's also pretty sensible to boot a small Linux system that can then analyze the full USB stick using all its file system drivers and offer a list of Linuxes to boot from that Linux (using e.g. kexec).

There's been multiple "multi-boot enablers" that can scan your storage for .iso files and just let you select which one to boot. Many of these are bitrotting (the days where admins where running around with USB sticks to boot or install something on client PCs are over; netboot has gotten pretty easy, and modern OSes all have sensible remote access and update capabilities. In the server business, having board management controllers that let you just boot a file that you upload through a web console are standard.); honestly, I haven't used one of these in more than 5 years.

Most of these use(d) GRUB to do the actual booting; the most common (and probably still best-supported) one is Ventoy. You'll need to install ventoy to your stick once (and unnecessarily, but ultimately, most easily, destroys the data on your stick, unless you don't use the automated installation and figure out how to do it manually into your existing stick);

./Ventoy2Disk.sh -i -g /dev/sdYOURUSBSTICK

does the job.

Afterwards, you just drop iso files into the exfat partition (you can actually also put them somewhere else and manually browse the other partitions from GRUB).

You can do whatever you want with the other partition(s) on the stick.