8

I'm using a Mac. In Bash, I'm trying to decode a base64 string and then try to print the Hex value. I'm using base64 -d command and then assigning it to a variable.

myText='YYN29+2wV2XRAHymIyhgytWuqY4atgHnIUFfXA7FPOA='
myTextBytes=$(echo -n "$myText" | base64 --decode)
echo -n $myTextBytes | xxd -p -c 99999

The output of the above script is:

618376f7edb05765d17ca6232860cad5aea98e1ab601e721415f5c0ec53ce0

However, if i were to run the following command directly:

echo -n "$myText" | base64 --decode | xxd -p -c 100000

I get:

618376f7edb05765d1007ca6232860cad5aea98e1ab601e721415f5c0ec53ce0

I even tried using openssl enc -base64 and I get the same result. That is, 00 is getting deleted when assigning to a variable. How do I preserve the 00 when I'm assigning it to a variable?

2
  • 6
    Did you get "bash: warning: command substitution: ignored null byte in input" as part of the output of your 3-line script? What does 'bash --version' tell you? The bash provided with a Mac is an older version, I think. My version 4.4.20(1) gives me that warning.
    – Wastrel
    Aug 26, 2023 at 15:51
  • I'm on GNU bash, version 3.2.57(1)-release (arm64-apple-darwin22). That's why i didn't see the warning. Now, it makes sense why it went missing without any warning
    – SilleBille
    Aug 28, 2023 at 17:56

1 Answer 1

14

That's not "00" as in a string of two 0x30 characters (ASCII "0"), but a NUL character, 0x00. You can see this if you pipe the output of the decode command into od -a or od -x.

Bash's handling of command substitutions (like var1=$( command )) strips out NUL/0x00 characters. However, since Bash 4.4, it also prints out a warning. Inability to handle NUL bytes is a limitation of most shells, zsh being an exception. However, there are issues with passing 0x00 characters embedded within arguments to commands that zsh is still subject to (because it's not an issue within zsh but rather in the exec() family of calls to invoke commands).

If you don't want to switch to another language like Perl, Python, Ruby, etc., then I suggest trying zsh.

4
  • 4
    At least (modern) bash gives a warning for that case; dash also strips and ksh93 truncates. While a shell can be written so that a var can contain NUL when used inside the shell, like zsh, shell vars are often (mostly?) used to pass as an argument and/or environment variable to program(s) you run -- and the OS calls that do that are defined to use C-language strings which are terminated (truncated) by NUL. Aug 26, 2023 at 4:47
  • 4
    Beware that $(...) also strips trailing newline characters (0x0A bytes), so even in zsh, you'd have to work around it (with the usual var=$(print -r - $encoded | base64 -d; print .); var=${var%.}) Aug 26, 2023 at 5:33
  • 17
    One of the reasons base64 is commonly used in the first place is to transmit and protect binary data through non-binary safe paths. Probably your best bet (assuming you stick with bash) is to keep it as base64 until you are ready to use it and then decode it and use it in a pipe (as worked in your second command).
    – user10489
    Aug 26, 2023 at 7:35
  • 2
    zsh handles the NUL here fine, the commands they gave work directly and give the output with the tell-tale 00 in the middle. No need to doubt that part, though the other caveats mentioned above do apply. In particular, echo $x works with NUL bytes with zsh's built-in echo, but any external implementation of echo won't and can't work like that.
    – ilkkachu
    Aug 26, 2023 at 21:25

This site is temporarily in read-only mode and not accepting new answers.

Not the answer you're looking for? Browse other questions tagged .