4

I found bash ignores binary zero on input when reading using the read buildin command. Is there a way around that?

The task is reading from a pipe that delivers binary data chunks of 12 bytes at a time, i.e. 2 ints of 16 bit and 2 ints of 32 bit. Data rate is low, performance no issue. Since bash variables are C-style, the obvious read -N 12 struct does not work, bytes beyond a NUL are not accessible. So I figured I need to read the data byte by byte, using read -N 1 byte. Problems easy to fix are escapes (requires -r), and UTF multi-byte character coding (export LC_ALL=C). The problem I'm so far unable to solve is to deal with zero bytes. I thought they'd show up as empty variable byte, but in fact read -r -N 1 byte does not return at all upon zero (ignores zeros) but returns with the next following non-zero byte in the data stream.

This is what I was attempting to do, which, as long as no zero comes in, works without flaws:

export LC_ALL=C

while true;
  do
     for ((index = 0; index < 12; index++))
       do
          read -r -N 1 byte
          if [ -n "${byte}" ]; then
               struct[${index}]=$(echo -n "${byte}" | od -An -td1)
             else
               struct[${index}]=0
            fi
       done
... # some arithmetics reconstructing the four bitfields and processing them
  done < pipe

It turns out the else branch in the if is never taken. A data chunk of 12 bytes that includes a zero does not make the for loop run 12 times, instead it awaits more data to fill the struct array. I demonstrated the behaviour by feeding the pipe 12 bytes using the command

echo -en "ABCDE\tGH\0JKL" > pipe

Since it is so easy to fool oneself with this, I verified the sending of zeros with

~# mkfifo pipe
~# od -An -td1 <pipe &
[1] 25512
~# echo -en "ABCDE\tGH\0JKL" > pipe
~#    65   66   67   68   69    9   71   72    0   74   75   76

[1]+  Done                    od -An -td1 < /root/pipe

Is there a way to change this behaviour of bash? Or how else can the zero bytes be read?

4
  • Bash simply can't store NUL bytes in variables. You may want to look to other languages.
    – Kusalananda
    Commented Dec 29, 2020 at 14:40
  • AFAIK, zsh is the only shell that can store 0 bytes in its variables. ksh93 has some helper features to help you base64 encode/decode data. Commented Dec 29, 2020 at 15:04
  • I do not need the NUL byte stored in a variable. As I pointed out, bash uses C-style strings, so in a sense, every variable contains a NUL byte. That's why I expect to receive an empty string from the read builtin. So all I need is that the read builtin returns on every byte, not non-zero ones only. Commented Dec 29, 2020 at 15:29

1 Answer 1

3

bash variables can't store NUL bytes (only zsh does, though see also ksh93's printf %B and typeset -b using base64 encoding). Its read builtin will also skip NUL bytes in input.

However, here, you could use:

LC_ALL=C IFS= read -rd '' -n1 c

That is read up to one byte off a NUL-delimited record. So if $c is empty, that means either EOF (but then read's exit status would be non-zero) or a NUL byte was read.

For both, you can get the numeric value of that byte with:

LC_ALL=C printf -v value %d "'$c"

So:

while
  IFS= LC_ALL=C read -rd '' -n1 c &&
    LC_ALL=C printf -v value %d "'$c"
do
  echo "Got byte with value $value"
done

Would read the input one byte at a time until EOF and support NUL bytes.

Or you could always do:

value=$(dd bs=1 count=1 2> /dev/null | od -An -vtu1)

Or with some od implementations:

value=$(od -N1 -An -vtu1)

Though that implies forking extra processes and run external executables (and if stdin is a terminal device, that will not put it out of icanon mode like read does).

2
  • 1
    Wow. Using -n, rather than -N as the read builtin argument indeed nailed it. I had not tried -n because -N seemed indeed so much better suited to my purpose. I had tried -d '', though, without any effect, which is understandable given that the specs of -N clearly state that no delimiters are observed. Now, with -n I do have to care about delimiters, but that's easy (as between the lines you point out too). Thanks a lot! Commented Dec 29, 2020 at 16:33
  • I take it you suggest using printf```` instead of od``` because it's a builtin and therefore should be faster, as no new process needs to be forked? Commented Dec 29, 2020 at 16:40

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .