11

I'm trying to find the first non-zero byte (starting from an optional offset) on a block device using dd and print its offset, but I am stuck. I didn't mention dd in the title as I figured there might be a more appropriate tool than dd to do this, but I figured dd should be a good start. If you know of a more appropriate tool and/or more efficient way to reach my goal, that's fine too.

In the meantime I'll show you how far I've come with dd in bash, so far.

#!/bin/bash

# infile is just a temporary test file for now, which will be replaced with /dev/sdb, for instance
infile=test.txt
offset=0

while true; do
  byte=`dd status='none' bs=1 count=1 if="$infile" skip=$offset`
  ret=$?

  # the following doesn't appear to work
  # ret is always 0, even when the end of file/device is reached
  # how do I correctly determine if dd has reached the end of file/device?
  if [ $ret -gt 0 ]; then
    echo 'error, or end of file reached'
    break
  fi

  # I don't know how to correctly determine if the byte is non-zero
  # how do I determine if the read byte is non-zero?
  if [ $byte ???? ]; then
    echo "non-zero byte found at $offset"
    break
  fi

  ((++offset))
done


As you can see, I'm stuck with two issues that I don't know how to solve:
a. How do I make the while loop break when dd has reached the end of the file/device? dd gives an exit code of 0, where I expected a non-zero exit code instead.
b. How do I evaluate whether the byte that dd read and returns on stdout is non-zero? I think I've read somewhere that special care should be taken in bash with \0 bytes as well, but I'm not even sure this pertains to this situation.

Can you give me some hints on how to proceed, or perhaps suggest and alternative way to achieve my goal?

3
  • 2
    It is not a good idea to read a block device one byte at a time - read a block into a file with dd then either read the file a byte at a time or perhaps use something like cut to detect the null (zero) byte(s) otherwise known as \n. Might be necessary to prefill your file with non-null bytes. Jun 1 at 14:11
  • 1
    Reading past the end of a file isn't an error, it just gives you nothing, since there's nothing there. With the command substitution, you get the single byte in $byte, except if it's a newline (which command substitution removes), or a NUL (which Bash treats as end of string). In those cases you'd get an empty variable, and I don't think there's any way to tell them apart in just Bash.
    – ilkkachu
    Jun 1 at 14:24
  • 1
    Also... this is the same as using a shell loop to process text, you're running at least one invocation of dd per byte of input, so it'll be horribly slow. Well, (apart from the issues with NUL and newline), you could read larger blocks and process the string one byte at a time, but really, you'd be better off just taking an actual programming language more suited for data processing.
    – ilkkachu
    Jun 1 at 14:29
30

You can do this using cmp, comparing to /dev/zero:

cmp /path/to/block-device /dev/zero

cmp will give you the offset of the first non-zero byte.

If you want to skip bytes, you can use GNU cmp’s -i option, or if you’re not using GNU cmp, feed it the appropriate data using dd:

cmp -i 100 /path/to/block-device /dev/zero
dd if=/path/to/block-device bs=1 skip=100 | cmp - /dev/zero

This will work with any file, not just block devices.

2
  • 6
    Very clever! See also cmp < /path/to/block-device <#((100)) - /dev/zero in ksh93 (or the equivalent with sysseek in zsh) for those not constrained to using bash. Jun 1 at 14:30
  • 2
    this is indeed very clever -- finally a useful usage for /dev/zero! Jun 1 at 21:13
8

Stephen Kitt's answer makes this a bit pointless (it is more concise and more than one order of magnitude faster), but an alternative you have is to (hex)dump the content of your device, one byte per line, and pipe it to a program that prints the address of the first byte whose representation is not 00 and exits as soon as it finds it:

od -Ad -w1 -tx1 /dev/device | awk '$2 && $2 != "00" { print $1 + 1; exit }'

od's -j option allows you to optionally select the number of bytes to skip (at the beginning of the input).

A much faster variation (thanks to Peter Cordes' comments) requires a bit more of typing:

od -Ad -tx1 | awk '
  {
    for (i=2; i<=NF; i++)
      if ($i != "00") {
        print ($1 + i -1)
        exit
      }
  }'

Allowing od to output data in its preferred format requires to compute the offset of the first non-zero byte by adding its position in the line it appears on to the line's address.

8
  • 1
    It may well be that awk is mainly what is slowing this down. grep might be a bit faster: od -Ad -w1 -tx1 /dev/sda | grep -E -m1 -v '\*|00$' Jun 2 at 0:27
  • 1
    @DigitalTrauma I wouldn't know. On my system, grep, awk and sed take roughly the same time (either a few ten milliseconds with a real device and a few seconds with a testing file starting with some hundred MiB of nulls); they just read three short lines at most, after all. I preferred awk mainly because POSIX grep doesn't support -m.
    – fra-san
    Jun 2 at 8:56
  • 1
    I typically use hexdump -C /dev/device | less for this, because hexdump collapses repeats (at least of 00 lines) by default, so no need to burn CPU time formatting into hex and parsing the 00s. That makes it usable even if there's a non-zero header, although cmp -l foo /dev/zero | less also handles that case. But anyway, hexdump -C isn't amazingly efficient perhaps, compared to how fast it's possible for modern CPUs to scan memory for non-zero bytes/words, but it's usable for gigabyte-sized files. Jun 2 at 10:55
  • 1
    (time hd /dev/zero (and control-C) shows most of the cost is user time, like 2.0 user vs. 0.168 sys, so that's pretty bad. time cmp /dev/zero /dev/full shows a 50:50 split of user/kernel time, so it's scanning (err comparing :/) memory about as fast as copy_to_user can copy, presumably using memcmp so it's taking advantage of glibc's optimized hand-written asm loop using AVX2 SIMD instructions (on my Skylake CPU)). Jun 2 at 11:02
  • 2
    @PeterCordes od collapses repeated values by default too (I chose it because it's POSIX; though I see hexdump is probably faster). It isn't unbearably slow because the goal, here, allows for limiting the emitted lines to three at most.
    – fra-san
    Jun 2 at 11:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.