7

So I'd like to pass the first 512 bytes of binaryFile.dd as the second parameter to myProgram but bash strips out all the NUL chars. Is there any way to avoid this in bash or am I on a hiding to nothing?

myProgram parameter1 "$(head -c 512 binaryFile.dd)"
  • or pass those bytes to a named pipe that your program can open & read from – Jeff Schaller Feb 14 '17 at 12:02
6

There is no way to pass a null byte in the parameter of a command. This is not because of a limitation of bash, although bash has this limitation as well. This is a limitation of the interface to run a command: it treats a null byte as the end of the parameter. There's no escaping mechanism.

Most shells don't support null bytes in variables or in the arguments of functions and builtins. Zsh is a notable exception.

$ ksh -c 'a=$(printf foo\\0bar); printf "$a"' | od -t x1
0000000 66 6f 6f
0000003
$ bash -c 'a=$(printf foo\\0bar); printf "$a"' | od -t x1
0000000 66 6f 6f 62 61 72
0000006
$ zsh -c 'a=$(printf foo\\0bar); printf "$a"' | od -t x1
0000000 66 6f 6f 00 62 61 72
0000007

But even with zsh, if you attempt to pass a parameter to an external command, then anything following a null byte is ignored — not by zsh but by the kernel.

$ zsh -c 'a=$(printf foo\\0bar); /usr/bin/printf "$a"' | od -t x1
0000000 66 6f 6f
0000003

If you want to pass null bytes to a program, you need to find some way other than a command line parameter.

head -c 512 binaryFile.dd | myProgram --read-parameter2-from-stdin parameter1
myProgram --read-parameter2-from-file=<(head -c 512 binaryFile.dd) parameter1
| improve this answer | |
  • What is removing the nuls in bash is the "command expansion", not the kernel. – Isaac Feb 15 '17 at 1:03
  • @sorontar That's what's removing the nuls if you use a command expansion, yes. I never claim otherwise. It is, however, not the reason why what technicalbloke is trying to do can't be made to work. Zsh copes with null bytes just fine but that still doesn't allow passing a null byte as a parameter. – Gilles 'SO- stop being evil' Feb 15 '17 at 12:22
  • The question is about bash limitations, not the kernel: "Using binary data as a parameter in bash". That the kernel also has (c induced) limitations is interesting but not what was asked. – Isaac Feb 15 '17 at 12:42
  • @sorontar bash can't get around limitations the kernel has in spawning processes and passing parameters. – Barmar Feb 15 '17 at 17:48
  • @Barmar I never said it did. – Isaac Feb 16 '17 at 1:46
3

bash is not best suited to handle binary data directly.

Either use the binary data with a file, either use the hexadecimal string representing your data.

To convert in hexadecimal you can use hexdump, xxd, od.

For example to convert a 512 bytes to an hexadecimal string, use

xxd -ps -c 512 file.bin

To convert it back to binary use

echo "$myhexstring" | xxd -r -ps > file.bin
| improve this answer | |
  • This doesn't address the problem in the question, which is to pass a null in a command line parameter. – Gilles 'SO- stop being evil' Feb 14 '17 at 23:21
1

No, there is no way in which a string in bash could contain a NUL (\0).
Therefore, a variable (as it contains a string) could not contain a NUL.

The reason is that bash is written with the c paradigm of "a string ends in a NUL".[1] The linux kernel also impose such limitation.[2] But even if the kernel were to allow NULs in strings[3] (arguments), most shells, and bash in particular, could not include NULs inside variables [4].

Positional parameters ($1, $2, etc.) are equivalent to variables and can also not contain NULs.

However, nuls could exist in files, in streams, and in printf:

$ printf 'test\0nuls\n' | od -vAn -tx1c
  74  65  73  74  00  6e  75  6c  73  0a
   t   e   s   t  \0   n   u   l   s  \n

As you can see, printf creates a NUL and it flows thru the pipe (|). But NULs are striped from "command executions":

$ echo $(printf 'test\0nuls\n') | od -vAn -tx1c
bash: warning: command substitution: ignored null byte in input
  74  65  73  74  6e  75  6c  73  0a
   t   e   s   t   n   u   l   s  \n

In bash 4.4 it even emits a warning. zsh in this case, silently replace the NULs by an space:

$ zsh -c ' echo $(printf "test\0nuls\n") | od -vAn -tx1c'
  74  65  73  74  20  6e  75  6c  73  0a
   t   e   s   t       n   u   l   s  \n

We can create a file that contains NULs with printf and either cat, head, tail or dd part of the file which includes the NULs:

$ printf 'test\0nuls\0in\0files\0\n' > testnul.bin
$ cat testnul.bin | xxd -ps
74657374006e756c7300696e0066696c6573000a

$ head -c 7 testnul.bin | xxd -ps
74657374006e75

$ dd if=testnul.bin bs=7 count=1 | xxd -ps
74657374006e75
1+0 records in
1+0 records out
7 bytes copied, 0.000655689 s, 10.7 kB/s

$ dd if=testnul.bin bs=7 count=1 2>/dev/null| xxd -ps
74657374006e75

In your case, there is no simple[5] way to have the contents of a binary file as an argument. Maybe the hex representation could work:

$ myProgram "$parameter1" "$(xxd -ps -c 512 binaryFile.dd)"

Thanks to @Gilles for all the additional work (and detail) below.

1

[1] All comes down to the old definition of a C string that «strings end in a NUL (\0)». This paradigm has been coded in several C libraries and tools, of which, POSIX has several examples. Like strcpy in here that states (emphasis mine):

The strcpy() function shall copy the string pointed to by s2 (including the terminating NUL character) into the array pointed to by s1.

That means that it is assumed that a string is terminated by a NUL.
Or, in other words, there could be only one NUL, the last one.

2

[2] The execve() system call, also defined in POSIX, expects that strings (command arguments) end in a NUL. That's why even shells that could work with NULs (most don't, with the notable exception of zsh):

$ zsh -c 'a=$(printf "included\0null"); printf "$a"' | od -vAn -tx1c
  69  6e  63  6c  75  64  65  64  00  6e  75  6c  6c
   i   n   c   l   u   d   e   d  \0   n   u   l   l

Can not use NULs in arguments passed by the execve() call:

$ zsh -c 'a=$(printf "included\0null"); /usr/bin/printf "$a"' | od -vAn -tx1c
  69  6e  63  6c  75  64  65  64
   i   n   c   l   u   d   e   d

3

[3] But even if the kernel where able to include NULs in arguments, bash will not allow them:

$ bash -c 'a=$(printf "included\0null"); /usr/bin/printf "$a"' | od -vAn -tx1c
bash: warning: command substitution: ignored null byte in input
  69  6e  63  6c  75  64  65  64  6e  75  6c  6c
   i   n   c   l   u   d   e   d   n   u   l   l

In bash 4.4 it even emits a warning when a NUL is removed.

4

[4] Most shells, and bash in particular, could not include NULs inside variables.

$ printf 'included\0null' | od -vAn -tx1c
  69  6e  63  6c  75  64  65  64  00  6e  75  6c  6c
   i   n   c   l   u   d   e   d  \0   n   u   l   l

$ printf 'included\0null' | ( read a; printf '%s\n' "$a" | od -vAn -tx1c )
  69  6e  63  6c  75  64  65  64  6e  75  6c  6c
   i   n   c   l   u   d   e   d   n   u   l   l

If the running shell is zsh, this (instead) will work with a null:

$ zsh -c 'printf "included\0null" | ( read a; printf "%s\n" "$a" | od -vAn -tx1c )'
  69  6e  63  6c  75  64  65  64  00  6e  75  6c  6c  0a
   i   n   c   l   u   d   e   d  \0   n   u   l   l  \n

5

[5] Meaning that a "direct" (simple) inclusion of a byte of value 0 (\0) is impossible. But an encoded (complex), either using C-string $'\0', in hex, base 64, or some equivalent, the value of zero could be included.

| improve this answer | |
  • 1
    A C program can use whatever data structure it wants to represent strings; null-termination is the usual way to do it, but far from the only way. The real limitation is that execve takes the argument list as an array of null-terminated strings, so whatever format the shell uses internally needs to be converted (and nulls lost) in order to actually pass it as an argument list. – Gordon Davisson Feb 15 '17 at 2:06
  • And the C program bash has chosen the C strings convention that strings are null-terminated. @GordonDavisson It is true that execve also has such limitation. But even if the exec family of functions didn't have such limitation, bash does. Bash use the C-string convention that strings are null-terminated and therefore can not have strings (and all variables) which contain any NUL. The question asks about bash. Answering only about execve hides half of the truth. – Isaac Feb 17 '17 at 22:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.