bitwise-OR 2 binary files

Question

I did 2 rescue attempts on a dying HDD a while back; I ran (GNU) ddrescue first, and then straight dd with manual seeking. I want to get the best of both images. Since any empty stretches in the files will just be 0's, bitwise-AND should be sufficient to merge the two files.

Is there a utility that allows me to create a file that is the OR of two input files?

(I am using ArchLinux, but I'm happy to install from source if it's not in the repos)

Answer 1

I don't know of a utility that does this, but it should be pretty easy to write a program to do this. Here's a skeletal example in python:

#!/usr/bin/env python
f=open("/path/to/image1","rb")
g=open("/path/to/image2","rb")
h=open("/path/to/imageMerge","wb") #Output file
while True:
     data1=f.read(1) #Read a byte
     data2=g.read(1) #Read a byte
     if (data1 and data2): #Check that neither file has ended
          h.write(chr(ord(data1) | ord(data2))) #Or the bytes
     elif (data1): #If image1 is longer, clean up
          h.write(data1) 
          data1=f.read()
          h.write(data1)
          break
     elif (data2): #If image2 is longer, clean up
          h.write(data2)
          data2=g.read()
          h.write(data2)
          break
     else: #No cleanup needed if images are same length
          break
f.close()
g.close() 
h.close()

Or a C program that should run faster (but is significantly more likely to have an unnoticed bug):

#include <stdio.h>
#include <string.h>

#define BS 1024

int main() {
    FILE *f1,*f2,*fout;
    size_t bs1,bs2;
    f1=fopen("image1","r");
    f2=fopen("image2","r");
    fout=fopen("imageMerge","w");
    if(!(f1 && f2 && fout))
        return 1;
    char buffer1[BS];
    char buffer2[BS];
    char bufferout[BS];
    while(1) {
        bs1=fread(buffer1,1,BS,f1); //Read files to buffers, BS bytes at a time
        bs2=fread(buffer2,1,BS,f2);
        size_t x;
        for(x=0;bs1 && bs2;--bs1,--bs2,++x) //If we have data in both, 
            bufferout[x]=buffer1[x] | buffer2[x]; //write OR of the two to output buffer
        memcpy(bufferout+x,buffer1+x,bs1); //If bs1 is longer, copy the rest to the output buffer
        memcpy(bufferout+x,buffer2+x,bs2); //If bs2 is longer, copy the rest to the output buffer
        x+=bs1+bs2;
        fwrite(bufferout,1,x,fout);
        if(x!=BS)
            break;
    }
}

Answer 2

Python

with open('file1', 'rb') as in1, open('file2', 'rb') as in2, open('outfile', 'wb') as out:
    while True:
        bytes1 = in1.read(1024)
        bytes2 = in2.read(1024)
        if not bytes1 or not bytes2:
            break
        out.write(bytes(b1 | b2 for (b1, b2) in zip(bytes1, bytes2)))

This is roughly 10× faster than Chris's Python solution due to reading 1024 bytes at a time. It also uses the with open pattern as this is more reliable at closing files (e.g. in case of error).

This seems to work for me with Python 3.6.3 (successfully merging 2 partial torrent files), but has not been thoroughly tested.

Perhaps the if ...: break pattern could be removed and instead use while in1 or in2:

Answer 3

This is not bitwise or, but it works for (whole blocks of) zeroes:

dd conv=sparse,notrunc if=foo of=baz
dd conv=sparse,notrunc if=bar of=baz

Due to sparse it skips writing anything that is zero in the source file.

So baz will look like bar, plus whatever was zero in bar but not zero in foo.

In other words if there's non-zero data that's not identical in foo and bar, bar wins.