1

An Stackoverflow answer says that command substitution removes NUL and trailing newline(s), and provides the following commands to prevent it from doing so,

FILE="$(mktemp)"
printf "a\0\n" > "$FILE"
S="$(uuencode -m "$FILE" /dev/stdout)"
uudecode -o /dev/stdout <(printf "$S") | od -tx1
rm "$FILE"

cat "$FILE" doesn't remove NUL or trailing newlines. Can it replace

S="$(uuencode -m "$FILE" /dev/stdout)"
uudecode -o /dev/stdout <(printf "$S") 

so that the suggested commands can be simplified to

FILE="$(mktemp)"
printf "a\0\n" > "$FILE"
cat "$FILE" | od -tx1
rm "$FILE"

?

  • cat "$FILE" doesn't remove NUL or trailing newlines, but $(cat "$FILE") would. – glenn jackman Nov 20 '18 at 21:02
  • I updated. I am not planning to use cat "$FILE" in a command substitution, and does it work now? – Ben Nov 20 '18 at 21:06
  • 1
    I'm starting to wonder what it is you're trying to do with the command substitution and the NULs in the end. Perhaps it would be possible to give better answers if the big picture was visible. – ilkkachu Nov 20 '18 at 21:13
  • @ilkkachu nothing in particular. Just wondering why the author used uuencode and uudecode instead of cat? – Ben Nov 20 '18 at 21:16
  • 1
    @Ben - the SO answer you referenced provides the explanation. Bash variables (S in that example) cannot contain null bytes so uuencode is needed to encode that null byte to something else – iruvar Nov 20 '18 at 21:22
3

The important thing about the linked question is in the title: "How to read a file into a variable in shell?" The answer is that you don't, not in Bash, since Bash can't handle the NUL bytes.

This can be worked around somewhat by encoding the file so that the literal NULs are represented some other way. That could be Base64 encoding (as uuencode -m does), or URL encoding, or whatever. That's what the answer there does, it encodes the file using uuencode to a "nicer" string with no NULs, then stores that in the variable S. (A plain S=$(cat file) would not work, it would try to store the file unchanged, with the NUL bytes and all.) The opposite action of decoding the file is also shown.

Usually, there's not much need for that, and one should probably try to avoid doing that. It's easier to process files by keeping them as files and processing them with external tools as necessary. But the question there was specifically about how to store a file (an arbitrary file) in a shell variable. In theory, there just might be some use for that. Maybe.


Though in your case, if you just want to print some fixed string, then printf "a\0\n" > $filename or similar works just fine since any byte can be represented with the \NNN escapes.

  • Thanks. I figure out what I want now. I want to perform substitution using parameter expansion echo -n "${string//"$pattern"/"$replacement"}", and assume it is impossible to do it other way. The value of string can only come from a file, but I don't know how to store the file content into the variable string: string="$(cat "$inputfile")" will remove NUL and trailing newline(s). What would you do? – Ben Nov 20 '18 at 23:04
  • In S="$(uuencode -m "$FILE" /dev/stdout)", is there some case where $S is useful? – Ben Nov 21 '18 at 0:00
  • @Ben, if you have a file with possible NULs, and you want to perform substitutions on it, use sed (or tr, or Perl, or maybe even awk, depending on what exactly you want to do). Base64 encoded data is really hard to process since you have the original bytes smashed into characters in groups. Some other sort of encoding would lend itself better for processing in the shell, but still, just use the tools made for the purpose... – ilkkachu Nov 21 '18 at 0:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.