Given a file, I need to search for the number of blocks that matches an input block $templateBlock of certain size $blockSize.

What is an efficient way to use dd to accomplish this task? The following method is bad and slow because it invokes dd too many times.

while [ $i -lt $totalBlocks]; do
  dd if="$pathToFile" bs=$blockSize count=1 skip=$i | diff $templateBlock -
  # Increase $count if $? is 0
  # Increase $i by 1 

The best way will probably be for dd to read $totalSize and for me to process only $blockSize at a time for all things read. This way, I only have to call dd once. In other words, have dd read all (or many) blocks at once, and I can process each block one at a time. Is it possible to do this?

Maybe something like this:

dd if="$pathToFile" bs=$blockSize | for-each-block {
  diff $templateBlock -
  # Update $count if $? is 0

Ideally, I want to use only shell utilities.
In other words, I will prefer not to have to write a program using Python or similar.


In a shell script, it might be better to use cmp from GNU diffutils. It compares data for you and can even skip offsets --ignore-initial=SKIP1:SKIP2 so you can run cmp for each sector offset, and it will exit on the first difference it finds... which is semi-efficient but still, that's a LOT of cmp calls if you want to run it for each sector...

You could also use grep -abo or strings -t d to find possible byte offset candidates for you, but that depends on the actual pattern you're looking for. The advantage for these would be one program call to search the whole thing as opposed to thousands of calls (one per sector).

Unless you happen to find a command line utility that just happens to do all the work for you, you can't beat a small C/Go/Python script that searches the whole thing in one go as per your requirements...

Your script idea of course works too but it just means calling dd (or read, or whatever) again inside the loop to read one block of stdin at a time, it's probably slower than before...

  • Thanks for the tips. I will probably end up writing a program then. – nehcsivart Apr 12 '16 at 2:35

You can use the split command to run a "filter" upon successive chunks of a file, and write a filter using cmp, echo, and true to gain a newline for every successful match, then use wc for counting these. For example, if $F is the file, $TB the template block file, and $SZ the size of the template block, then it could look like the following command line:

$ split -b $SZ --filter="cmp $TB >& /dev/null && echo ; true" $F | wc -l

Note the true that ensures the filter command line succeeds, and to save typing you could use : instead, although that doesn't improve readability.

Note also the redirection of all output from cmp, i.e., both its stdout and its stderr, since only its return code is of interest.

  • Thanks for the suggestion! This method is slightly faster than my current method for small size files. I'll see how much faster it is for my actual file after it is done. Thanks! – nehcsivart Apr 12 '16 at 5:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.