What command will give me a binary delta between two files and let me apply it?

Question

I have two large files.

• a.qcow2 is 16GB.

• b.qcow2 is 17GB.

The first 16GB of b.qcow2 is 95% identical to a.qcow2.

What Linux commands will let me produce the delta between the two files? (Which should be less than 2GB in size.)

What linux commands will let me apply the delta to a.qcow2 to produce output identical to b.qcow2?

Do the delta formats have safety measures, such as the source and target sizes and hashes?

Answer 1

If you are going to have two similar VMs what i recommend you is using qemu option: backing_file

qemu-img create -f qcow2 \
                -o backing_file=/path/to/base/image.qcow2 \
                /path/to/new_guest/image.qcow2 

What this allows you it's to have a master image and then only a differential file between the original and the seccond, saving you a lot of GB. Also if you really need to know the binary difference between this files i recommend you to checkout the following commands:

• rdiff
• dhex
• xdelta

Answer 2

I'm assuming you want to transport the difference between two systems, and this is why you can't rebase your QCOW images on a shared root.

The rsync tool will let you generate a binary difference. Here's a worked example with (considerably) smaller files

ls -l ?.qcow2
-rwxrwx--- 1 root root 76993536 Jan  2  2020 a.qcow2
-rwxrwx--- 1 root root 77337600 Oct 25 19:00 b.qcow2

As per your question, we need to get the differences from b.qcow2 to a.qcow2. Note that provided rsync can get to both source and destination there is no need for them to be on the same server

rsync -av --only-write-batch b_to_a.rsyncdiff b.qcow2 a.qcow2

This determines the difference and writes it to b_to_a.rsyncdiff, also creating a helper shellscript along the way

ls -l b_to_a*
-rw------- 1 root root 35617731 Oct 26 09:26 b_to_a.rsyncdiff
-rwx------ 1 root root       54 Oct 26 09:26 b_to_a.rsyncdiff.sh

cat b_to_a.rsyncdiff.sh
rsync -av --read-batch b_to_a.rsyncdiff ${1:-a.qcow2}

To apply the change you just need the original source a.qcow2 and the diff

rsync -av --read-batch b_to_a.rsyncdiff a.qcow2
receiving incremental file list
b.qcow2

sent 61,509 bytes  received 35,621,188 bytes  71,365,394.00 bytes/sec
total size is 77,337,600  speedup is 2.17

Result

ls -l ?.qcow2
-rwxrwx--- 1 root root 77337600 Oct 25 19:00 a.qcow2
-rwxrwx--- 1 root root 77337600 Oct 25 19:00 b.qcow2

Answer 3

Personally I'm a huge fan of deltup which is well known only among Gentoo users.

It's very fast and too often beats xdelta by a large margin.