Fetching individual bytes from a binary file into a variable with bash

Question

I need to make a fairly simple bash script to pull bytes one at a time from a binary file, send it out a serial port, and then wait for a byte to come back before I send the next one. This is effectively for an EEPROM programmer, where an off the shelf solution won't work.

I'm mostly stuck at pulling the bytes out of the file into a variable, beyond that I know I can echo the values down the serial port and read them back with dd. (Is there a better way to do that also?)

Thanks for the help!

Answer 1

Use dd or xxd (part of Vim), for example to read one byte (-l) at offset 100 (-s) from binary file try:

xxd -p -l1 -s 100 file.bin

to use hex offsets, in Bash you can use this syntax $((16#64)), e.g.:

echo $((16#$(xxd -p -l1 -s $((16#FC)) file.bin)))

which reads byte at offset FC and print it in decimal format.

Alternatively use dd, like:

dd if=file.bin seek=$((16#FC)) bs=1 count=5 status=none

which will dump raw data of 5 bytes at hex offset FC.

Then you can assign it into variable, however it won't work when your data has NULL bytes, therefore either you can skip them (xxd -a) or as workaround you can store them in plain hexadecimal format.

For example:

1. Read 10 bytes at offset 10 into variable which contain bytes in hex format:

   hex=$(xxd -p -l 10 -s 10 file.bin)
   

2. Then write them into file or device:

   xxd -r -p > out.bin <<<$hex
   

---

Here are few useful functions in Bash:

set -e

# Read single decimal value at given offset from the file.
read_value() {
  file="$1"
  offset="$2"
  [ -n "$file" ]
  [ -n "$offset" ]
  echo $((16#$(xxd -p -l1 -s $offset "$file")))
}

# Read bytes in hex format at given offset from the file.
read_data() {
  file="$1"
  offset="$2"
  count="$3:-1"
  xxd -p -l $count -s $offset "$file"
}

Sample usage:

read_value file.bin $((16#FC)) # Read 0xFC offset from file.bin.

Answer 2

I’m not sure I understand the big picture of what you’re asking for, but I have a couple of approaches that you might want to consider:

Use dd to read your input file

exec 3< input_file
while true
do
    dd bs=1 count=1 <&3 > this_byte
    if ! [ -s this_byte ]       # If size is zero, we’re done.
    then
        break
    fi
    code using the this_byte file
          ︙
done

Use od to digest your input file

od -bv input_file | while read a0 a1 a2 a3 a4 a5 a6 a7 a8 a9 a10 a11 a12 a13 a14 a15 a16
do                              # Skip $a0; it’s the address.
    for byte_val in "$a1" "$a2" "$a3" "$a4" "$a5" "$a6" "$a7" "$a8" "$a9" "$a10" \
                                "$a11" "$a12" "$a13" "$a14" "$a15" "$a16"
    do
        if [ "$byte_val" = "" ]
        then
            break
        fi
        code using the $byte_val value
        for example: echo -e "\0$byte_val\c"
              ︙
    done
done

Answer 3

This is probably easier to do with a tiny C program, or an (almost) one-liner in a scripting language like Python or Perl.

The Unix shell is an interpreter for user commands, that (for convenience in writing repetitive commands) has rudimentary facilities for programming. It often gets (mis)used because a version of the Bourne shell is guaranteed to be available on any Unix system, not because it is a particularly good (or even half-way decent) programming language.

Answer 4

You can use perl instead:

$ printf '%s%s%s' 'ăâé' | LC_ALL=C.UTF-8 perl -Mopen=locale -ne '
    BEGIN { $/ = \1 }
    printf "%s\n", $_;
'
ă
â
é