BTRFS issue - disappearing/reappearing contents? Two independent versions of filesystem depending on mounted subvolume

Question

Initial problem

So, I mounted a subvolume of a new BTRFS pool as /home. I thought I had copied everything to it before reformatting the old drive to add to the pool, but on later examination some files (all my multimedia) were missing. I resigned myself to rebuilding the collection.

Later on, I ran the built-in "Baobob" disk-usage utility. It showed more contents on /mnt/wd than expected. I looked more closely. /mnt/wd/home/$USER/multimedia still had all my "missing" files (and not what I'd put in ~/multimedia since the move) (root of the BTRFS system has label wd).

Looking back at ~/multimedia, the old files were not there and the new files still were.

---

Commonality between missing files

The files that were initially missing had one thing in common. Before being copied to the new drive, the files that didn't go missing were on an ext4 that was mounted as /home. The ones that did were all on an ntfs partition of the old drive, mounted as /media/$USER/data, with symlinks pointing to them from /home/$USER. That was to accomodate a dual-boot that I have since stopped using. When copying everything to the new volume (with rsync, in case it's relevant), I put all the files together in one subvolume instead of splitting them and symlinking (mv of the contents of one directory into the other, after both rsyncs had completed). I did fiddle about with subdirectories a bit before settling on that result, but I ended up with just the one subvolume as best as I can tell. Before mounting the new subvolume as /home, this has appeared to work. After, the view from /home seemed to match the original /home partition, and the view from /mnt/wd/home seemed to match the merged version.

---

Initial configuration/status

My fstab at this point was:

# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/nvme0n1p5 during installation
UUID=699ad207-3b40-4caa-9926-503830121327 /               ext4    errors=remount-ro 0       1
# BTRFS pool
UUID=24f6bbb6-cccb-4f12-808c-195f18b19e05   /home   btrfs   defaults,autodefrag,subvol=home 0   2
UUID=24f6bbb6-cccb-4f12-808c-195f18b19e05   /mnt/app_drive  btrfs   defaults,autodefrag,subvol=applications 0   2
UUID=24f6bbb6-cccb-4f12-808c-195f18b19e05   /mnt/snapshots  btrfs   defaults,autodefrag,subvol=snapshots    0   2
# Swap file
/swapfile1 none swap sw 0 0

I have also mounted the root of the BTRFS volume as /mnt/wd via sudo mount UUID=24f6bbb6-cccb-4f12-808c-195f18b19e05 /mnt/wd.

Running sudo btrfs subvolume list . from any of the mounted locations of any subvolume (or the root):

ID 638 gen 4492 top level 5 path home
ID 639 gen 4433 top level 5 path applications
ID 640 gen 4404 top level 5 path snapshots
ID 800 gen 4402 top level 640 path snapshots/home_2017-07-06

But ls ./$USER/multimedia (and any of the other missing directories) shows entirely different contents when run from /home or from /mnt/wd/home.

qwertystop@dt /home $ ls $USER/multimedia
pics  unprepared
qwertystop@dt /home $ cd /mnt/wd/home
qwertystop@dt /mnt/wd/home $ ls $USER/multimedia
3d  ml_resource  music  osu!  phone-ready  pics  Unprepared  vids

Does anyone have the faintest idea what's going on here, or how I might reconcile these two? I had the idea of just copying everything from one to the other - but that would still mean that I'd have to make sure to do that for every new file, best-case.

---

Further experimentation

Snapshots

I tried making snapshots. No matter which mount point I use for creating the snapshot, and no matter where I put it, it always matches the version found in /mnt/wd/home, not the version in /home. I've tried:

sudo btrfs subvolume snapshot /home /home/snapshot
sudo btrfs subvolume snapshot /mnt/wd/home /mnt/snapshots/snapshot
sudo btrfs subvolume snapshot /home /mnt/snapshots/snapshot

And all of them are identical. Which is a plus for internal consistency, except it doesn't tell me why the inconsistency still exists when I try to access any files.

---

fstab/mounting experimentation:

I have tried mounting the BTRFS root at /mnt/wd in my fstab (line is , placed before the /home line) instead of post-boot with mount. My fstab at this point was:

# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/nvme0n1p5 during installation
UUID=699ad207-3b40-4caa-9926-503830121327 /               ext4    errors=remount-ro 0       1
# BTRFS pool
UUID=24f6bbb6-cccb-4f12-808c-195f18b19e05   /mnt/wd   btrfs   defaults,autodefrag 0   2
UUID=24f6bbb6-cccb-4f12-808c-195f18b19e05   /home   btrfs   defaults,autodefrag,subvol=home 0   2
UUID=24f6bbb6-cccb-4f12-808c-195f18b19e05   /mnt/app_drive  btrfs   defaults,autodefrag,subvol=applications 0   2
UUID=24f6bbb6-cccb-4f12-808c-195f18b19e05   /mnt/snapshots  btrfs   defaults,autodefrag,subvol=snapshots    0   2
# Swap file
/swapfile1 none swap sw 0 0

This produced no change.

I have tried not mounting the home subvolume at /home in fstab at all (commenting the line out). When I then uncommented the line and mounted /home, the resulting mounted drive matched the one on /mnt/wd, with the old files and not the new ones. So I think this might be something going wonky at boot time?

"Unchanged" files still diverge

The files that didn't go missing between the two versions are still being tracked separately by the filesystem. I looked at tail .bash_history in each, and they were different.

Problem is with the mount point and the subvolume in combination?

I mounted the home subvolume at /mnt/foo (and retried, later, at /foo) instead of /home. It matches the old version. I made a snapshot of the home subvolume, called it newhome, and mounted that at /home. It matches the new version (missing old files, present new files).

I added a new file to one of the directories that was only in the new version (still in newhome). I changed back to mounting home at /home and rebooted. The file was gone. Changed back to using newhome, the file was still there. Did the same in the other order (putting it in the home version and checking newhome), and with putting the file in one of the directories that had not differed. The results were the same - the file only existed in the subvolume that it had been put in, and only in the new version (which only appeared when mounted as /home, but was different for each one). That rules out my newest idea, which was that it saw "the subvolume mounted at the /home mount-point" as a distinct thing from whatever subvolume I actually mounted, for whatever reason.

I took an unrelated subvolume, added a subdirectory for my username, and mounted it at /home. The result was that the OS still recognized my username/password, but regenerated all the config files and directories in it as though I were a new user - it did not use the "new version" of the /home partition.

I think at this point I can summarize the situation. This does not help me solve the problem or determine a cause.

tl;dr

The subvolume named "home", and snapshots descended from it, are actually two subvolumes, one of which only appears when it is mounted as /home, the other when it is mounted elsewhere or visible as a subdirectory of the root volume of the BTRFS device-pool. For unknown reason, the two diverged at some point no later than the first time I mounted it as /home, such that the version mounted at /home contained the contents of the old /home partition which I had rsync-ed over, and the version mounted anywhere else contained the filesystem as I had modified it after that initial rsync (the replacement of several symlinks-to-another-partition with the files previously linked).

I don't know why this happened. I can confirm that it does not happen when a subvolume not descended from "home" is mounted as /home, with no files from "home" copied in. I can confirm that it does happen with a small subset of files copied to the new subvolume. I will now test copying all files, with cp instead of making a snapshot.

That worked, with one tricky bit that may point to the problem. When I attempted to cp -a --reflink=always ./* from /home/qwertystop to my new home subvolume, I got an error message per-file: "Invalid cross-device link." This did not appear when running the same command relative to /mnt/wd/home/qwertystop. It would seem that the problem causes, is caused by, or otherwise relates to the computer reading one mount location as being on a different device than the other. This is, I suppose, technically possible - it's a BTRFS pool of three distinct devices.

When I switched to copying out of the /mnt/wd/home version, it seemed initially to work, but light use quickly revealed that the duplication appeared in the new subvolume the same as in the old.

Answer 1

I have solved the problem, though not the related questions of why it's a problem or where the initial divergence came from.

When I copied my home directory to a new subvolume, but did not copy /home/.ecryptfs, the duplication ceased. Thus, it seems that the duplication was caused somehow by .ecryptfs. I am uncertain whether this is a bug or expected behaviour of some system of which I am unaware.

Be aware that you have disabled the encryption your system was installed with.

ecryptfs is a stacking filesystem. It can be mounted at login time - i.e. at the point the user enters their password, which they want to encrypt the data with. findmnt (or mount) will show a filesystem of type ecryptfs mounted at /home/$USER. Files written to /home/$USER/ are encrypted by ecryptfs, and then stored somewhere in /home/.ecryptfs/$USER/ (i.e. btrfs). This is just one possible ecryptfs setup; there are other ways it could be arranged as well.

Please don't nit-pick the above description as I've never used it. You can find the community-provided documentation here:

https://help.ubuntu.com/community/EncryptedHome#Encrypted_Home

This should explain why you see some different files when you log in with the btrfs volume mounted at /home and look at your home directory /home/$USER, compared to the files you see in the $USER/ directory of the same btrfs volume when mounted at /mnt.

"Invalid cross-device link." This did not appear when running the same command relative to /mnt/wd/home/qwertystop. It would seem that the problem causes, is caused by, or otherwise relates to the computer reading one mount location as being on a different device than the other.

I'm afraid "cross-device link" is a misnomer. It is really a cross-filesystem link. You can't hard-link or ref-link files from the btrfs filesystem into the ecryptfs filesystem, or vice-versa.

I tried making snapshots. No matter which mount point I use for creating the snapshot, and no matter where I put it, it always matches the version found in /mnt/wd/home

You should now be able to see how this happens as well.

WARNING: Restoring snapshots or otherwise messing with the backing store while you're still logged in & have ecryptfs mounted, sounds very worrying. Instead, you could enable the root user or create a special admin user account with an unencrypted home directory, and use that when you need to restore a snapshot of /home.

Answer 2

I have solved the problem, though not the related questions of why it's a problem or where the initial divergence came from.

When I copied my home directory to a new subvolume, but did not copy /home/.ecryptfs, the duplication ceased. Thus, it seems that the duplication was caused somehow by .ecryptfs. I am uncertain whether this is a bug or expected behaviour of some system of which I am unaware.