Here are some thoughts - I am still learning this and will update this as I go.
How to choose the union filesystem
There are two ways to look at this:
How do the features of each one compare?
For some common use cases, which one should I choose?
I'll compare unionfs / unionfs-fuse / overlayfs / aufs / mergerfs, the latter being a replacement for mhddfs.
Backslash will escape it. Since the mount command sends it as-is (as can be seen with strace), this has to be the kernel which uses a backslash to escape it.
mount -t overlay \
-o 'lowerdir=/tmp/a\,b/lower,upperdir=/tmp/a\,b/upper,workdir=/tmp/a\,b/work' \
I think the kernel's escapes in octal seen in /proc/mounts is to help ...
Then is there some idea to change UID/GID on the fly by mount options?
Yes. Many file systems support uid=NNNN and gid=NNNN options at mount time. It depends on the file system whether that subsequently applies to all files and directories within the image.
mount -t fstype -o loop,uid=1234,gid=1234 /source/file.img /destination/point
lxc.pre.mount gets executed before the rootfs gets loaded:
lxc.hook.pre-mount = /var/lib/lxc/container0/mount-squashfs.sh
lxc.rootfs.path = overlayfs:/var/lib/lxc/container0/rootfs:/var/lib/lxc/container0/delta0
And in the mount script:
mount -nt squashfs -o ro /var/lib/lxc/container0/rootfs.sqsh /var/lib/lxc/container0/rootfs
You could use MergerFS for this. With this you need to mount your disks as you would do normally and use MergerFS to mount one on top of another.
Excellent podcast episode with the developer of MergerFS
On first glance it seems like I can only have one writable location and the other locations are just there to provide the files they have.
This is correct, OverlayFS only supports one writable layer at the top. As such, I'd say it's not really suitable for the use case you describe.
The goal is to make it so I can use all the storage as one giant ...