6

If I have a large file, I can use ln to make make "copies" of it that don't use up extra disk space. But what if I don't want an exact copy of the file? Is there a way to create a new version of a file with some modifications without needing to copy the whole file and use twice the disk space?

My motivation is editing id3 tags on mp3 files from a torrent download. I don't want to edit the downloaded files directly because that messes up the seeding but I also don't want to copy the files and use twice the disk space just to edit some id3 tags their headers.

  • As you mention ln does not really make a copy. It`d be better descriped as that it creates a second filename in the filesystem for the very same inode related data (being the file). – humanityANDpeace Oct 25 '14 at 11:53
  • Can those tags even be updated in place? Because if they cannot, then file system features won't help you here. A standard link (like you would create with ln) won't help you here. What you are looking for is snapshots with copy-on-write capabilities. It is not a standard feature, but some file systems have it. I am aware of such snapshots existing both for full file system as well as for individual files. – kasperd Oct 25 '14 at 22:49
6

If there is no built-in capability in the program that you use, to overlay new information in some way over a base file, you have to resolve this on the filesystem level, transparently to the application using the file. Because of your space requirement a revision control system would not suffice, although it provides you with multiple versions.

One thing you can investigate is to store the files on a Btrfs filesystem and have the originals in one "originals" snapshot and the updated versions in a view based on this snapshot. This should work well for ID3v1 tags (as they are the end of a file) and also for those files that have ID3v2 tags¹, as long as they have enough reserved space for the changes and do not require rewriting of the MP3 file. Thus only the actual blocks changed for a file are taking extra disk space.

If you add additional file in the originals you have to make an explicit cp --reflink src dst for all the files added at a later stage. Your downloads would then work with the originals and your id3 editor ( e.g. picard) and your music player on the derived view. Unchanged (or not yet changed) files in that view will look exactly the same as under the originals.

Example (starting with an Btrfs volume on /data0 and a test.mp3 file in /tmp):

/data0$ btrfs subvolume create /data0/mp3org
Create subvolume '/data0/mp3org'
/data0$ cp /tmp/test.mp3 mp3org/
/data0$ btrfs subvolume snapshot /data0/mp3org/ /data0/id3update
Create a snapshot of '/data0/mp3org/' in '/data0/id3update'

The file test.mp3 is now available in both directories (mp3org and id3update):

/data0$ ls -l /data0/mp3org
total 7600
-rw-rw-r-- 1 avanderneut users 7781043 Oct 25 15:21 test.mp3
/data0$ ls -l /data0/id3update/
total 7600
-rw-rw-r-- 1 avanderneut users 7781043 Oct 25 15:21 test.mp3

Change the one in the snapshot:

/data0$ id3v2 -c "This is a change" id3update/test.mp3
/data0$ ls -l /data0/mp3org
total 7600
-rw-rw-r-- 1 avanderneut users 7781043 Oct 25 15:21 test.mp3
/data0$ ls -l /data0/id3update/
total 7608
-rw-rw-r-- 1 avanderneut users 7781043 Oct 25 15:24 test.mp3

The file size has not changed, but the contents of the second one did. This is because the comment did fit in the reserved space for the id3v2 in the original file.

/data0$  grep -F "is a change" mp3org/* id3update/*
Binary file id3update/test.mp3 matches

Copy another file in the original subvolume, it doesn't show up in id3update:

/data0$ cp /tmp/test.mp3 mp3org/abc.mp3
/data0$ ls -l mp3org/ id3update/
id3update/:
total 7600
-rw-rw-r-- 1 avanderneut users 7781043 Oct 25 15:24 test.mp3

mp3org/:
total 15200
-rw-rw-r-- 1 avanderneut users 7781043 Oct 25 15:28 abc.mp3
-rw-rw-r-- 1 avanderneut users 7781043 Oct 25 15:21 test.mp3

Make an explicit reflink copy:

/data0$ cp --reflink mp3org/abc.mp3 id3update/
/data0$ ls -l mp3org/ id3update/
id3update/:
total 15200
-rw-rw-r-- 1 avanderneut users 7781043 Oct 25 15:29 abc.mp3
-rw-rw-r-- 1 avanderneut users 7781043 Oct 25 15:24 test.mp3

mp3org/:
total 15200
-rw-rw-r-- 1 avanderneut users 7781043 Oct 25 15:28 abc.mp3
-rw-rw-r-- 1 avanderneut users 7781043 Oct 25 15:21 test.mp3

And change the new file:

/data0$ id3v2 -c "another file change" id3update/abc.mp3
/data0$ grep -F change mp3org/* id3update/*
Binary file id3update/abc.mp3 matches
Binary file id3update/test.mp3 matches

If mp3org gets filled automatically you can keep id3update up to date by running a script on a regular basis that does the cp --reflink src dst if the destination doesn't yet exists.

¹which are most often at the beginning of a file

  • sweet. a cp --reflink --parents combo looks particularly promising. Thanks, Anthon. – mikeserv Oct 25 '14 at 13:58
0

Using a Version Control System such as git you can "check in" or "commit" any number of versions of your file. Git is probably one of the easiest Version Control Systems to get to grips with, because it doesn't require the use of a separate server; all the work can be done on your own local machine,and there is plenty of tutorial information available on the web. There are also various graphical front ends available for it. Git will store the different file versions internally as a compressed version of the original file and of the differences between versions. As part of the commit process you provide a message which briefly describes the state of the committed file(s), and afterwards you can browse the available versions and "check out" whichever version you currently want to work with.

  • 2
    As I already indicated in my answer, a version control system is inappropriate. It would have at least 2 times the file size requirement of the original and i3dedited version of the checked out files, plus a compressed copy (in the repository). The OP explicitly states that he doesn't want to use twice the disc space just to edit a tag. – Anthon Oct 25 '14 at 14:16
0

To find the difference there is command diff - compare files line by line. You can store differences to diff-file and at any time can apply it by patch - apply a diff file to an original. But in any way you should first to make a second file to make changes (you can delete it after receive diff-file).

If diff can't work with binary files even with --text parameter, you are free to use bsdiff or, possible, rdiff (thanks Anko for comment).

  • 1
    Brain dump: diff detects binary files and switches to summary mode, printing just Binary files X and Y differ, which is supremely unhelpful for patching. You can force it to treat the files as text with --text, so it prints the full listing. Those diffs apply cleanly with patch, though you'll want the --binary flag to disable line-ending conversion. Better-compressing binary patch tools exist though. (Feel free to edit these into your answer.) – Anko Oct 25 '14 at 17:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.