8

Using just shell tools, how can a binary stream containing NULLs (0x00 chars) be edited keeping the 0x00 chars in the output stream ?

The edit needs to replace a char in a specified position for another char (in the following example by the char '|'), like as:

dd ibs=1 skip=$offset count=$reglen status=none if=$ARQ |
        sed 's/./\|/2' |
        sed 's/./\|/5' #| more replacements....

But sed is removing all '\0x00' chars before the replacement.

EDIT - Demonstration of sed behavior in my environment using the @George Vasiliou test:

$ echo -e "lineA\nlineB\nlineC" | tr '\n' '\0' | od -t x1
0000000 6c 69 6e 65 41 00 6c 69 6e 65 42 00 6c 69 6e 65
0000020 43 00
0000022

$ echo -e "lineA\nlineB\nlineC" | tr '\n' '\0' | sed 's/./|/5' | od -t x1
0000000 6c 69 6e 65 7c 6c 69 6e 65 42 6c 69 6e 65 43
0000017

My environment is an AIX 7.1 and the sed that is there isn't the gnu version.

9

sed is a text utility. It works with text lines (sequences of non-NUL characters (not bytes) of limited length delimited by a newline character).

If you want to change the 2nd and 5th byte of a sequence of bytes, it won't work for several reasons:

  • sed works on text. If the input contains NUL characters, doesn't end in a newline character, has more than LINE_MAX bytes in between two newline characters, contains sequences of bytes that don't form valid characters, depending on the sed implementation, it won't work at all. (note that GNU sed doesn't have many of those limitations).
  • even if that binary input happens to form valid text, . matches characters, not bytes, so may match more than one byte.
  • because the sed code is run for every line of the input, that would change the second and fifth character of each line, not of the whole input.

To treat input as arbitrary arrays of bytes (without the NUL byte limitation, or length limitations), you may want to use perl instead:

 dd.... | perl -0777 -pe 'for $o (1, 4) {substr($_, $o, 1) = "|"}'

Example:

$ printf 'a\0b\0cd' |
>   perl -0777 -pe 'for $o (1, 4) {substr($_, $o, 1) = "|"}' |
>   od -Ax -tx1 -tc
000000  61  7c  62  00  7c  64
         a   |   b  \0   |   d
000006

Or you could use an intermediate text representation, like using vim's xxd helper:

dd... | xxd -p | sed '1s/../7c/2;1s/../7c/5' | xxd -p -r

xxd -p gives a hex dump with 60 characters per line by default. Above we're replacing the second and fifth 2-digit hex of the first line with 7c, the number for ASCII |.

  • Thanks. I was building a workaround using xxd. Great ! Both solutions worked in AIX. – Luciano Feb 20 '17 at 14:00
1

Are you sure ? with a simple test this doesn't seems to happen in my case (gnu sed 4.2.2)

$ echo -e "lineA\nlineB\nlineC"
lineA
lineB
lineC
$ echo -e "lineA\nlineB\nlineC" |tr '\n' '\0'
lineAlineBlineC
$ echo -e "lineA\nlineB\nlineC" |tr '\n' '\0' |sed 's/./|/5'
line|lineBlineC
# Verification if the nulls are still there:
$ echo -e "lineA\nlineB\nlineC" |tr '\n' '\0' |sed 's/./|/5' |tr '\0' '\n'                                                                                                
line|
lineB
lineC

With further testing, null will be lost if you replace the 6th character in my tests (null position):

$ echo -e "lineA\nlineB\nlineC" |tr '\n' '\0' |sed 's/./|/6' |tr '\0' '\n'
lineA|lineB 
lineC

$ echo -e "lineA\nlineB\nlineC" |tr '\n' '\0' |sed 's/./|/7' |tr '\0' '\n'
lineA
|ineB           
lineC 
  • @Luciano See update – George Vasiliou Feb 20 '17 at 12:44
  • Look at my edit – Luciano Feb 20 '17 at 12:46
  • @Luciano , i also tried with sed --posix which acc to my manual disables all GNU extensions, but still null bytes are present.... – George Vasiliou Feb 20 '17 at 13:00
  • I tried sed in Linux, and yes looks to be working. But I need to get it working in AIX. – Luciano Feb 20 '17 at 13:56
  • 1
    @Luciano , Sure, i can understand this... Unfortunatelly i do not have AIX to help you, and as far as i know it seems that there are not AIX Shells online to play with... I'm sure the answer of Mr Chazelas will help you. – George Vasiliou Feb 20 '17 at 14:04
0

Try bbe - sed clone for binary streams: https://sourceforge.net/projects/bbe/

  • Could you add some supporting detail, such as how the user in their AIX environment might use it? Also, note that the question says "Using just shell tools", so they may be restricted from compiling/installing additional tools, – Jeff Schaller Mar 18 '18 at 15:49
  • Are you sure you are linking to the right tool? Your link goes to a "Block Based Encryption aka 2Bx4Bx2B" project last updated in 2013 – Ale Jan 30 at 15:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.