I wanted to see what files are added on top of ISO 9660 when LiveUSB Linux is running. When booted with persistence upper and work folders are on USB drive clearly seen. I run mount on Linux booted from LiveUSB "usual way" (w/out persistence) and saw / is mounted via overlayfs and upperdir=/cow/upper. But sudo ls /cow gives no such file or directory.

Where is /cow and how to see its contents?

Added 1: I was able to extract contents of initrd from liveUSB via unmkinitramfs (see https://unix.stackexchange.com/a/495524/446998)

$ find . -type f -exec bash -c 'cat {} | grep "/cow/upper" && ls -l {}' \;
    if [ ! -d /cow/upper ]; then
        mkdir -p /cow/upper
            /cow/lost+found|/cow/upper|/cow/log|/cow/crash|/cow/install-logs-*)     continue ;;
            mv "$cow_content" /cow/upper
    mount -t overlay -o "upperdir=/cow/upper,lowerdir=$mounts,workdir=/cow/work" "/cow" "$rootmnt" || panic "overlay mount failed"
-rw-r--r-- 1 alex alex 33834 Jun 24  2020 ./main/scripts/casper  

Next step I envision is to understand how /cow is created because is not see in contents of initrd

3 Answers 3


// Experience based on kubuntu 22 lts livecd

after chroot

the last step of ramdisk (/cdrom/casper/initrd) is
run-init {rootmnt}" "${init}" "$@"

which do something like
chroot {rootmnt}" "${init}" "$@"

after that step it may affect the observation of the original mount point.

before chroot

Fortunately there are ways to pause at an interactive shell before chrooting (press ctrl+d or exit to continue)

kernel boot cmdline args
may do the trick.

//BTW: manjaro 22 only support break=premount (same as break=y) or break=postmount, not support multiple values compound with ,

also another cmdline args might help
debug or debug=y
which turn on detail log during ramdisk running

// boot args can be edit at grub menu by press e
// those information comes from reading the ramdisk script

in ramdisk

you already unmkinitramfs and found scripts/casper
which handle casper-rw persistence things


setup_overlay() {

    # Mount up the writable layer, if it is persistent then it may well
    # tell us what format we should be using.
    mkdir -p /cow

    # Looking for "$(root_persistence_label)" device or file
    if [ -n "${PERSISTENT}" ]; then
        cowprobe=$(find_cow_device "$(root_persistence_label)")
        if [ -b "${cowprobe}" ]; then
            cow_fstype=$(get_fstype "${cowprobe}")
            [ "$quiet" != "y" ] && log_warning_msg "Unable to find the persistent medium"

    mount -t ${cow_fstype} -o ${cow_mountopt} ${cowdevice} /cow || panic "Can not mount $cowdevice on /cow"

code here determine the persistence partition, and mount it

keep /cow after chroot

in ramdisk shell, after /cow /root ready

mkdir /root/_cow
mount -o bind /cow /root/_cow

then after chroot into system, /_cow can access the original outer /cow

  • 2
    Thanks! I've discovered solution with -bind myself before reading your answer, I've posted an answer with sed command to edit casper script. As for pausing with kernel parameters, this is an interesting thought. BTW, please read my answer and comment why original mounts for showmounts result in empty folders, if you know. TIA Jan 8 at 5:09
  • Yes, On-site investigation is simple and effective
    – yurenchen
    Jan 10 at 15:23
  • @Martian2020 I didn't understand exactly the situation you described, the variable and part of script. Maybe you can explicitly point out which path should not be empty before you replace move by bind
    – yurenchen
    Jan 10 at 15:28

LiveUSB sticks use an overlay file system such as overlayfs. There are a couple of different ones so I'm not certain which it uses but the principle is pretty much the same in each.

The root file system that you see is composed of layers. For a LiveUSB this will be

  • an upper writable layer
  • a read-only lower layer containing your operating system image and often stored in a squashfs file.

Initially you see just the files in the lower layer because the upper (writable) layer is empty. When you write new files they will be written to the upper layer. But other changes such as moving a file and editing a file will actually cause the file to be copied to the upper layer from the lower layer.

The upper layer can be a few things. If the LiveUSB doesn't have persistence it's often a tempfs stored only in RAM. However if there's persistence it may be something different... perhaps it's a writable file formatted with EXT4.

If it's a writable file then you may be able to find that file and mount it separate to the overlayfs to poke around what's inside.

Where is /cow and how to see its contents?

That's the most tricky part of this question to answer. The fact is it may not exactly exist anymore, at least not under that name.

It did exist during the execution of initramfs. But then initramfs called pivot_root which would have moved it... and then most likely tried to unmount it so now no userland process can actually see it any more. The Kernel must logically still have it mounted though.

If it was mounted with persistence it may be possible to discover exactly what /cow was and re-mount it elsewhere.

  • thank you some details added. As you saw in the question I've already dived into initrd so I hoped to be able to keep some handle to access it later (by editing inittird scripts). Dec 4, 2021 at 0:31
  • 1
    @Martian2020 I'm trying to get a working LiveUSB so I can test this... I have a sneaking suspicion that after pivot_root it's /bin/init (systemd) that unmounts "new_root". A working theory I'd like to test is to edit the grub boot entry to run /bin/bash instead of /bin/init... that may give you chance to see what's actually mounted there. Dec 4, 2021 at 0:33
  • I've booted with persistence and changes remain on USB (writable/upper), but mount does not show USB folders, it is same /cow/upper, so I guess it is under the hood. Mounting another overlay over /usb/writable/upper did not change a thing - the data continued to appear in original folder only, no changes observed in new upper Dec 4, 2021 at 0:34
  • I meant overlay with /usb/writable/upper as lowerdir Dec 4, 2021 at 0:36
  • @Martian2020 Yeah the "layers" in an overlayfs are actually file systems not devices. It's more like a bind mount. So the game is figuring out what was mounted to /cow Dec 4, 2021 at 0:36

I've discovered this before the answer my @yurenchen, which gives this idea at the end.

Below particular command edits casper script adding bind mount for shownmounts boot option. One need to unpack and repack initramfs (initrd file for Linux Mint 21) for that. After that /cow's layers are visible in /casper/cow after liveUSO system completed boot.

# add mount of /cow
sudo sed --in-place --regexp-extended -- 's|(if [[] -n "[$][{]SHOWMOUNTS[}]" []]; then)|\1\n\n        mkdir -p "${rootmnt}/${LIVE_MEDIA_PATH}/cow"\n        mount --bind /cow "${rootmnt}/${LIVE_MEDIA_PATH}/cow"\n|' $work_path/initrd/main/scripts/casper

# change mount option from "move" to "bind".
sudo sed --in-place --regexp-extended -- 's|(mount )-o move( "[$][{]d[}]" "[$][{]rootmnt[}]/[$][{]LIVE_MEDIA_PATH[}]/[$][{]d[#][#][*]/[}]")|\1--bind\2|' $work_path/initrd/main/scripts/casper

Also, as a bonus second command changes type of mount as for some reason in booted system in /casper I saw folder for squashfs, but it was empty. With with command it was not, but filled with contents. P.S. /rofs has same, but for only one upper layer, there could be several squashfs files in the distro's ISO.

  • after boot (chroot) I can see /cdrom/casper/filesystem.squashfs is here, and it already mount to /rofs/ well. sorry I didn't understand the situation you descripted. tried on mint 21 livecd.
    – yurenchen
    Jan 10 at 15:42
  • @yurenchen. filesystem.squashfs folder is empty in my case for default casper script. Is it not empty on your system? Jan 15 at 10:00
  • yes, It's there, I'm using linuxmint-21.1-cinnamon-64bit.iso, but with ventoy tool (in order to try multi distros in one usb, and with persistent, and customize injection to grub.cfg and ramdisk in runtime, it's really powerful)
    – yurenchen
    Jan 15 at 16:40
  • @yurenchen, "yes, It's there," indeed it is. But what does it contain? I've just booted LM 21 via Ventoy UEFI, replacing in default menu item with ctrl-e quiet splash with showmounts and /cdrom/casper/filesystem.squashfs is empty. No persistence flag(s) added, just that replacement. Jan 17 at 5:28

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .