1

Is it possible (in classical ext4, and/or in any other filesystem) to create two files that point to the same content, such that if one file is modified, the content is duplicated and the two files become different? It would be very practical to save space on my hard drive.

Context: I have some heavy videos that I share on an owncloud server that can be modified by lot's of people and therefore it may be possible that some people modify/remove these files... I really would like to make sure I have a backup of these files, and therefore I need for now to maintain two directories, the normal nextcloud one, and one "backup" directory, which (at least) double the size required to store it.

I was thinking to great a git repo on top of the nextcloud directory, and it make the backup process much easier when new videos are added (just git add .), but git still double the space, between the blob and the working directory.

Ideally, a solution that can be combined with git would be awesome (i.e. that allows me to create a history of the video changes, with commits, checkouts... without doubling the disk space.

Moreover, I'm curious to have solution for various file systems (especially if you have tricks for filesystems that do not implement snapshots ). Note that LVM snapshot is not really a solution as I don't want to backup my full volume, only some specific files/folders.

Thanks!

  • I think a robust way is to have a master directory tree, where only you have write access. Other people can use symbolic links to those files. When they want to modify a file, let them create a copy in some other directory. You can browse that directory at regular intervals, and if you wish, upgrade the version in the master directory tree. – sudodus Sep 17 '19 at 18:18
  • @sudodus : thanks for your answer. However, that looks way to complicated to explain to everybody in the association (I'm already having pain to let them use the online interface, and I'm kind of leaving the association). And still, I'm not even sure nextcloud provides way to do complicated links between two repositories that would map to two folders in my own computer... Or maybe I missed something? – tobiasBora Sep 17 '19 at 19:23
  • You can always use cp --reflink=always in btrfs without using snapshots. – Stéphane Chazelas Sep 18 '19 at 5:28
  • How many video files are there, and how many members of this association, that can read and edit files? – sudodus Sep 18 '19 at 12:55
4

Yes on a Copy On Write file systems (Btrfs, ZFS). git-annex is as close as you are likely to get on ext4. Note that you can mount --bind a LVM backed volume or a Btrfs file system over a folder in another file system.

  • It's funny to see it's not possible, it seems so easy to implement ^^ (after I don't know the details of file systems) I thought that at least it would be possible on btrfs or zfs. But your trick with bind mount is interesting, and git-annex looks super interesting, I'm reading more about it! (I'm waiting a bit more before accepting the answer in case someone comes with other interesting projects) – tobiasBora Sep 17 '19 at 19:27
1

Is it possible (in classical ext4, and/or in any other filesystem) to create two files that point to the same content, such that if one file is modified, the content is duplicated and the two files become different? It would be very practical to save space on my hard drive.

Hard links do that, provided that anyone who edits the files does it by creating a new file and renaming that on top of the original. That would break the link, since the new file is linked to only one name, and the other link gets overwritten by the rewrite.

The trouble is, that it's hard to know in general if a program is going to overwrite the data in the same inode, or if it's going to create a new file when saving.

Making the "backup copies" read-only might help, as then they couldn't be modified, but the links could still be removed and the same name recreated. But that might just lead to a lot of errors when the programs couldn't write to that inode.

So, the way I see it, file-level deduplication like that through hard links is possible, but only if you control the programs writing to the links.

0

I did a 'mini-test' and needed no symbolic links. It is enough to play with ownership and permissions in an ext4 file system. Let users read all video files, but write only their own files (including edited versions of other user's files).

Write permission of a file decides if it can be overwritten. Write permission of the directory decides if a file can be created and deleted. So the users should write their files in their own directories, for example in the Video directory. In addition to the standard tools and methods, the users should get some help to find the files of other users in a convenient way, I think with a graphical user interface (PCmanFM has a nice finder, but there might be better alternatives).

The system manager can decide if files should turn into master files owned by the system manager, which files to backup, when to backup and how to backup.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.