5

Is it possible (in classical ext4, and/or in any other filesystem) to create two files that point to the same content, such that if one file is modified, the content is duplicated and the two files become different? It would be very practical to save space on my hard drive.

Context: I have some heavy videos that I share on an owncloud server that can be modified by lot's of people and therefore it may be possible that some people modify/remove these files... I really would like to make sure I have a backup of these files, and therefore I need for now to maintain two directories, the normal nextcloud one, and one "backup" directory, which (at least) doubles the size required to store it.

I was thinking of creating a git repo on top of the nextcloud directory, and it make the backup process much easier when new videos are added (just git add .), but git still doubles the space, between the blob and the working directory.

Ideally, a solution that can be combined with git would be awesome (i.e. that allows me to create a history of the video changes, with commits, checkouts... without doubling the disk space).

Moreover, I'm curious to have a solution for various file systems (especially if you have tricks for filesystems that do not implement snapshots). Note that LVM snapshot is not really a solution as I don't want to backup my full volume, only some specific files/folders.

Improve this question
4
  • I think a robust way is to have a master directory tree, where only you have write access. Other people can use symbolic links to those files. When they want to modify a file, let them create a copy in some other directory. You can browse that directory at regular intervals, and if you wish, upgrade the version in the master directory tree.
    – sudodus
    Commented Sep 17, 2019 at 18:18
  • @sudodus : thanks for your answer. However, that looks way to complicated to explain to everybody in the association (I'm already having pain to let them use the online interface, and I'm kind of leaving the association). And still, I'm not even sure nextcloud provides way to do complicated links between two repositories that would map to two folders in my own computer... Or maybe I missed something?
    – tobiasBora
    Commented Sep 17, 2019 at 19:23
  • 1
    You can always use cp --reflink=always in btrfs without using snapshots.
    – Stéphane Chazelas
    Commented Sep 18, 2019 at 5:28
  • How many video files are there, and how many members of this association, that can read and edit files?
    – sudodus
    Commented Sep 18, 2019 at 12:55

3 Answers 3

Reset to default
6

Yes on a Copy On Write file systems (Btrfs, ZFS). git-annex is as close as you are likely to get on ext4. Note that you can mount --bind a LVM backed volume or a Btrfs file system over a folder in another file system.

Improve this answer
3
  • It's funny to see it's not possible, it seems so easy to implement ^^ (after I don't know the details of file systems) I thought that at least it would be possible on btrfs or zfs. But your trick with bind mount is interesting, and git-annex looks super interesting, I'm reading more about it! (I'm waiting a bit more before accepting the answer in case someone comes with other interesting projects)
    – tobiasBora
    Commented Sep 17, 2019 at 19:27
  • 1
    @tobiasBora It IS possible on btrfs, which the answer says. Why did you say "not possible"?
    – endolith
    Commented Jan 13, 2021 at 16:05
  • Yes, not sure why I wrote this comment ^^' I'm using ZFS for that now, and it works nicely!
    – tobiasBora
    Commented Jan 29, 2022 at 8:28
3

I guess the other answers don't say this explicitly, so yes, it's possible on btrfs using reflinks instead of hardlinks.

  • Hardlinks are two separate names for the same file, which occupies the same chunk of disk.
  • Reflinks are two different files, which happen to point to the same chunk of disk.

When a hardlink is modified, the other linked file also shows the modification, since both are referencing the same file.

When a reflink is modified, the other file is not modified. The edits are written to a new chunk of disk, while the unchanged parts of the two files still refer to the same chunks of disk, saving space.

Unlike a hard link where two directory entries refer to the same inode entry, with reflinks there are two inode entries, and it is the data blocks that are shared.

Link type Disk space
consumption		 Speed Editable Use across
file systems
partitions
Reflinks ✅ ✅ ✅ ❌
Symlinks ✅ ✅ ❌ ✅
Hard links ✅ ✅ ❌ ❌
File copy ❌ ❌ ✅ ✅

(source)

Improve this answer
2

Is it possible (in classical ext4, and/or in any other filesystem) to create two files that point to the same content, such that if one file is modified, the content is duplicated and the two files become different? It would be very practical to save space on my hard drive.

Hard links do that, provided that anyone who edits the files does it by creating a new file and renaming that on top of the original. That would break the link, since the new file is linked to only one name, and the other link gets overwritten by the rewrite.

The trouble is, that it's hard to know in general if a program is going to overwrite the data in the same inode, or if it's going to create a new file when saving.

Making the "backup copies" read-only might help, as then they couldn't be modified, but the links could still be removed and the same name recreated. But that might just lead to a lot of errors when the programs couldn't write to that inode.

So, the way I see it, file-level deduplication like that through hard links is possible, but only if you control the programs writing to the links.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .