Editing binary streams containing '\x00' bytes

Question

Using just shell tools, how can a binary stream containing NULLs (0x00 chars) be edited keeping the 0x00 chars in the output stream ?

The edit needs to replace a char in a specified position for another char (in the following example by the char '|'), like as:

dd ibs=1 skip=$offset count=$reglen status=none if=$ARQ |
        sed 's/./\|/2' |
        sed 's/./\|/5' #| more replacements....

But sed is removing all '\0x00' chars before the replacement.

EDIT - Demonstration of sed behavior in my environment using the @George Vasiliou test:

$ echo -e "lineA\nlineB\nlineC" | tr '\n' '\0' | od -t x1
0000000 6c 69 6e 65 41 00 6c 69 6e 65 42 00 6c 69 6e 65
0000020 43 00
0000022

$ echo -e "lineA\nlineB\nlineC" | tr '\n' '\0' | sed 's/./|/5' | od -t x1
0000000 6c 69 6e 65 7c 6c 69 6e 65 42 6c 69 6e 65 43
0000017

My environment is an AIX 7.1 and the sed that is there isn't the gnu version.

Answer 1

sed is a text utility. It works with text lines (sequences of non-NUL characters (not bytes) of limited length delimited by a newline character).

If you want to change the 2^nd and 5^th byte of a sequence of bytes, it won't work for several reasons:

• sed works on text. If the input contains NUL characters, doesn't end in a newline character, has more than LINE_MAX bytes in between two newline characters, contains sequences of bytes that don't form valid characters, depending on the sed implementation, it won't work at all. (note that GNU sed doesn't have many of those limitations).
• even if that binary input happens to form valid text, . matches characters, not bytes, so may match more than one byte.
• because the sed code is run for every line of the input, that would change the second and fifth character of each line, not of the whole input.

To treat input as arbitrary arrays of bytes (without the NUL byte limitation, or length limitations), you may want to use perl instead:

 dd.... | perl -0777 -pe 'for $o (1, 4) {substr($_, $o, 1) = "|"}'

Example:

$ printf 'a\0b\0cd' |
>   perl -0777 -pe 'for $o (1, 4) {substr($_, $o, 1) = "|"}' |
>   od -Ax -tx1 -tc
000000  61  7c  62  00  7c  64
         a   |   b  \0   |   d
000006

Or you could use an intermediate text representation, like using vim's xxd helper:

dd... | xxd -p | sed '1s/../7c/2;1s/../7c/5' | xxd -p -r

xxd -p gives a hex dump with 60 characters per line by default. Above we're replacing the second and fifth 2-digit hex of the first line with 7c, the number for ASCII |.

Answer 2

Are you sure ? with a simple test this doesn't seems to happen in my case (gnu sed 4.2.2)

$ echo -e "lineA\nlineB\nlineC"
lineA
lineB
lineC
$ echo -e "lineA\nlineB\nlineC" |tr '\n' '\0'
lineAlineBlineC
$ echo -e "lineA\nlineB\nlineC" |tr '\n' '\0' |sed 's/./|/5'
line|lineBlineC
# Verification if the nulls are still there:
$ echo -e "lineA\nlineB\nlineC" |tr '\n' '\0' |sed 's/./|/5' |tr '\0' '\n'                                                                                                
line|
lineB
lineC

With further testing, null will be lost if you replace the 6th character in my tests (null position):

$ echo -e "lineA\nlineB\nlineC" |tr '\n' '\0' |sed 's/./|/6' |tr '\0' '\n'
lineA|lineB 
lineC

$ echo -e "lineA\nlineB\nlineC" |tr '\n' '\0' |sed 's/./|/7' |tr '\0' '\n'
lineA
|ineB           
lineC 

Answer 3

Try bbe, a sed-like editor for binary streams.

Man page

bbe is a sed-like editor for binary files. Instead of reading input in lines as sed, bbe reads arbitrary blocks from an input stream and performs byte-related transformations on found blocks.