11

I'm trying to do some tricks with dd. I thought it would be possible to store some hexvalues in a variable called "header" to pipe it into dd.

My first step without a variable was this:

$ echo -ne "\x36\xc9\xda\x00\xb4" |dd of=hex
$ hd hex

00000000  36 c9 da 00 b4                                    |6....|
00000005

After that I tried this:

$ header=$(echo -ne "\x36\xc9\xda\x00\xb4") 
$ echo -n $header | hd

00000000  36 c9 da b4                                       |6...|
00000004

As you can see I lost my \x00 value in the $header variable. Does anyone have an explanation for this behavior? This is driving me crazy.

  • I get bash: warning: command substitution: ignored null byte in input. – Kusalananda Feb 24 '17 at 17:29
  • You are missing quotes it should be header="$(echo -ne "\x36\xc9\xda\x00\xb4")"; echo -n "$header" | hd however this just gives same result. – ctrl-alt-delor Feb 24 '17 at 19:01
  • This works header="\x36\xc9\xda\x00\xb4"; echo -n "$header" | hd, but is not the same thing as it is storing the human readable form. – ctrl-alt-delor Feb 24 '17 at 19:03
16

You can't store a null byte in a string because Bash uses C-style strings, which reserve the null byte for terminators. So you need to rewrite your script to simply pipe the sequence that contains the null byte without Bash needing to store it in the middle. For example, you can do this:

printf "\x36\xc9\xda\x00\xb4" | hd

Notice, by the way, that you don't need echo; you can use Bash's printf for this an many other simple tasks.

Or instead of chaining, you can use a temporary file:

printf "\x36\xc9\xda\x00\xb4" > /tmp/mysequence
hd /tmp/mysequence

Of course, this has the problem that the file /tmp/mysequence may already exist. And now you need to keep creating temporary files and saving their paths in strings.

Or you can avoid that by using process substitution:

hd <(printf "\x36\xc9\xda\x00\xb4")

The <(command) operator creates a named pipe in the file system, which will receive the output of command. hd will receive, as its first argument, the path to that pipe—which it will open and read almost like any file. You can read more about it here: https://unix.stackexchange.com/a/17117/136742.

  • 1
    While correct, this is an implementation detail and not the exact reason. I looked at it, and the POSIX standard actually requires this behaviour, so there you have the actual reason. (As some have pointed out, zsh will do it, but only in nōn-POSIX mode.) I actually looked into it because I was wondering if it was worth to implement this in mksh – mirabilos Feb 24 '17 at 19:33
  • @mirabilos, would you care to expand on that? AFAICT, behaviour is unspecified per POSIX for command substitution when the output has NUL characters, and for zsh in POSIX mode, the only relevant difference I can think of is that in sh emulation, \0 is not in the default value of $IFS. echo "$(printf 'a\0b')" still works OK in sh emulation in zsh. – Stéphane Chazelas Feb 24 '17 at 21:48
  • 3
    @mirabilos Considering that the shells predates the POSIX standard by a decade or more, I guess you could find out that the actual actual reason is that shells used C-style strings and the standard was built around that. – giusti Feb 25 '17 at 1:49
  • I found a good Q for detailed discussion on printf versus echo. unix.stackexchange.com/questions/65803/… – Paulb Feb 25 '17 at 12:34
8

You can use zsh instead which is the only shell that can store the NUL character in its variables. That character even happens to be in the default value of $IFS in zsh.

nul=$'\0'

Or:

nul=$'\x0'

Or

nul=$'\u0000'

Or

nul=$(printf '\0')

However note that you can't pass such a variable as an argument or environment variable to a command that is executed as the arguments and environment variables are NUL-delimited strings passed to the execve() system call (a limitation of the system's API, not the shell). In zsh, you can however pass NUL bytes as arguments to functions or builtin commands.

echo $'\0' # works
/bin/echo $'\0' # doesn't
  • 1
    "You can use zsh instead". No thanks - I'm teaching myself bash-scripting as a beginner right now. I don't want to confuse myself with an other syntax. But thank you veray much for suggest it – Frank Feb 24 '17 at 18:53
  • As a matter of fact, you used zsh syntax in your question. echo -n $header to mean to pass the content of the $header variable as a last argument to echo -n is zsh (or fish or rc or es) syntax, not bash syntax. In bash, that has a very different meaning. More generally zsh is like ksh (bash, the GNU shell, being more or less a part-clone of ksh, the Unix de-facto shell) but with most of the design idiosyncrasies of the Bourne shell fixed (and a lot of extra features, and a lot more user-friendly/less astonishing). – Stéphane Chazelas Feb 24 '17 at 20:03
  • Be careful: zsh may change a zero byte sometimes: echo $(printf 'ab\0cd') | od -vAn -tx1c prints ` 61 62 20 63 64 0a`, that is an space where a NUL should exist. – sorontar Feb 24 '17 at 20:58
  • 1
    And that is something no other (none, nil) shell will reproduce. That makes an script behave in very special ways in zsh. In my opinion: zsh is just trying to be too clever. – sorontar Feb 24 '17 at 21:17
  • 2
    Having "fixed" the design misfeatures present in the POSIX sh standard that getting accustomed to writing zsh scripts means one is getting accustomed to practices which would be buggy if exercised in any other shell. This isn't such a problem with a syntax that's so unlike a different language that skills or habits aren't likely to transfer, but such is not the case at hand. – Charles Duffy Feb 24 '17 at 21:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.