Update block device with image while writing only the Delta, is it possible?

Question

I use dd extensively as a means of software configuration control. The images are typically deployed on flash disk to update devices. I find I am often making small incremental updates to the image files and having to then re-copy the entire image over to the block device. This is fairly time consuming as images are typically 8GB in size. To compound the issue the images (once assembled) are not in an easily mountable format. In other words, making the change directly on the block is not possible.

I am trying to determine if there is a method to compare an image file to a block device and only update the blocks which require an update. I suspect this would be much faster than writing the entire disk for what probably equates to a 10kb delta in the image file.

Answer 1

The following is a fast hack for a small C program able to compare two files (file1, file2) blockwise and, if the blocks are different, copies the corresponding block from file1 to file2. Works for files and block devices as well. Do with it what you want, but at your own risk!

/*
Small program to blockwise compare two files and write different
blocks from file1 to file2.

Arguments: file1, file2, blocksize in bytes
If blocksize is not given, it is set to 512 (minimum)

No error checking, no intensive tests run - use at your own risk! 

*/

#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#include <string.h>

int main(argc, argv)
int argc;
char *argv[];
{

  char *fnamein;                  /* Input file name */
  char *fnameout;                 /* Output file name */
  char *bufin;                    /* Input buffer */
  char *bufout;                   /* Output buffer */
  int bufsize;                    /* Buffer size (blocksize) */
  int fdin;                       /* Input file descriptor*/
  int fdout;                      /* Output file descriptor*/
  int cnt;                        /* Current block # */

  /* Argument processing */

  if (argc < 3 || argc > 4) {
    fprintf(stderr,"Usage: %s infile outfile [bufsize]\n", argv[0]);
    exit(1);
  }

  fnamein = argv[1];
  fnameout = argv[2];
  if (argc == 4) {
    bufsize = atoi(argv[3]);
    if (bufsize < 512) {
      fprintf(stderr,"Error: Illegal value for [bufsize]: %s\n", argv[3]);
      exit(1);
    }
  } else {
    bufsize = 512;
  }

  fprintf(stderr, "Copying differing blocks from '%s' to '%s', blocksize is %i\n", fnamein, fnameout, bufsize);

  if (! ((bufin = malloc(bufsize)) && (bufout = malloc(bufsize)))) {
    fprintf(stderr,"Error: Can't allocate buffers: %i\n", bufsize);
    exit(1);  
  }
  fdin = open(fnamein, O_RDONLY);
  if (fdin < 0) {
    fprintf(stderr,"Error: Can't open input file: %s\n", fnamein);
    exit(1);  
  }

  fdout = open(fnameout, O_RDWR | O_SYNC);
  if (fdout < 0) {
    fprintf(stderr,"Error: Can't open ouput file: %s\n", fnameout);
    exit(1);  
  }

  cnt = 0;
  while (read(fdin, bufin, bufsize) == bufsize) {
    if (read(fdout, bufout, bufsize) == bufsize) {
      if (memcmp(bufin, bufout, bufsize) != 0) {
        fprintf(stderr, "Differing blocks at block # %i; writing block to %s\n", cnt, fnameout);
        if (lseek(fdout, -bufsize, SEEK_CUR) > -1) {
          if (write(fdout, bufin, bufsize) != bufsize) {
            fprintf(stderr,"Error: Unable to write to output file %s block # %i\n", fnameout, cnt);
            exit(1);
          }
        } else {
          fprintf(stderr,"Error: Unable to seek to output file %s block # %i\n", fnameout, cnt);
          exit(1);
        }
      }
    } else {
      fprintf(stderr,"Error: Unable to read from ouput file %s block # %i\n", fnameout, cnt);
      exit(1);
    }
    cnt++;
  }

  exit(0);
}

Answer 2

Look into qemu-img, using the original as a backing file, and creating a differencing disk in qcow2 format.

Access it as a block device using qemu-nbd (to convert it to an nbd device).

This will record the deltas into the qcow2 differencing disk and leave the original alone. The differencing disk grows by small values for each changed block.

To then apply those deltas to one or more "originals", use the "commit" qemu-img operation.

Answer 3

Well, as mentioned in a comment on your question, dd parameters bs, seek, skip and count are your friends. In addition, I would logically divide your images into a list of suitably-sized chunks (say, 10Mb each) and maintain the md5sum of each chunk (or a longer hash, if you fear collisions, which I wouldn’t). When a new image comes along, you only need to check the new image against the hashes (practically halfing the comparison time) and copy to disk only the changed chunks. You might even discover that some of your chunks are identical (all zeros, probably), and perform further optimizations accordingly.