4

I want to perform shuf --zero-terminated on multi-line strings with here-document.

  • 2
    I don't think you can do this in a here-document in either of those shells. Shells are generally not good with nulls (except zsh). Why do you want a here-document specifically? – Michael Homer Mar 1 at 4:54
  • @MichaelHomer Because I can't pass the multi-line strings entity(i.e string include the newline character) to shuf as one single entity. – illiterate Mar 1 at 5:02
7

Here-documents in Bash and dash don't support this. You can't store a null in a variable, they are removed from command substitutions, you can't write one in literally, and you can't use ANSI-C quoting inside the here-document. Neither shell is null-friendly and they are generally treated as (C-style) string terminators if one does get in.

You have a few options: use a real file, use zsh, use process substitution, or use standard input.


You can do exactly what you want in zsh, which is much more null-friendly.

zsh% null=$(printf '\x00')
zsh% hexdump -C <<EOT
heredoc> a${null}b${null}
heredoc> EOT
00000000  61 00 62 00 0a                                    |a.b..|
00000005

Note though that heredocs have an implicit terminating newline, which may not be desirable (it'll be an extra field for shuf after the final null).


For Bash, you can use process substitution almost equivalently to your heredoc in combination with printf or echo -e to create nulls inline:

bash$ hexdump -C < <(
printf 'item 1\x00item\n2\x00'
)
00000000  69 74 65 6d 20 31 00 69  74 65 6d 0a 32 00        |item 1.item.2.|
0000000e

This is not necessarily entirely equivalent to a here-document, because those are often secretly put into real files by the shell (which matters for seekability, among other things).

Since you probably want to suppress terminating newlines, you can't even use a heredoc internally within the commands there - it has to be printf/echo -ne if safe to get fine-grained control over the output.


You can't do process substitution in dash, but in any shell you could pipe in standard input from a subshell:

dash$ (
printf 'item 1\x00'
printf 'item\n2\x00'
) | hexdump -C
00000000  69 74 65 6d 20 31 00 69  74 65 6d 0a 32 00        |item 1.item.2.|
0000000e

shuf is happy to read from standard input by default, so that should work for your concrete use case as I understand it. If you have a more complex command, being on the right-hand side of a pipeline can introduce some confounding elements with scoping.


Finally, you could write your data into a real file using printf and use that instead of a here-document. That option has been covered in the other answer. You'll need to make sure you clean up the file afterwards, and may want to use mktemp or similar if available to create a safe filename if there are any live security concerns.

6

Thank you all. Let I post one answer base on you all and maybe best for me.

This script works nicely in bash and dash, no require real file or process substitution in bash, no require an extra slow external program call, even you don't need to worry about any escape problem in entities as %s in C printf, but you should still take care to string escape in your shell itself.

#!/bin/sh
printf '%s\0' "[tag1]
key1=value1
key2=value2
[/tag1]
" "[tag2]
key3=value3
key4=value4
[/tag2]
" | shuf --zero-terminated
#also see man printf(1)

For shuf only(not intent to general here-document alternative):

shuf --echo "[tag1]
key1=value1
key2=value2
[/tag1]" "[tag2]
key3=value3
key4=value4
[/tag2]"
  • 5
    Don't use echo for that, not even /bin/echo (on GNU systems for instance, whether /bin/echo supports -e depends on whether $POSIXLY_CORRECT is in the environment or not, on many other systems, /bin/echo doesn't support -e or -n or \x00). Use printf '%s\0' 'first record' 'second record'... | shuf... – Stéphane Chazelas Mar 1 at 8:35
3

I do not think you can do what you want in a heredoc. However it is trivial to do using echo as the following example shows:

$ cat demo
#!/bin/bash

echo -ne "one\0" > outfile
echo -ne "two\0" >> outfile
echo -ne "three\0" >> outfile

$ ./demo
$ od -a outfile
0000000   o   n   e nul   t   w   o nul   t   h   r   e   e nul
0000016
$ 
  • 1
    Thank you, but echo is not trivial, see my answer – illiterate Mar 1 at 8:16
  • @fpmurphy In addition to Stephane's excellent comment on illiterate's answer, see also the APPLICATION USAGE and RATIONALE sections of the POSIX spec for echo. – Charles Duffy Mar 1 at 17:04
  • @CharlesDuffy. If you do not like to use echo, simply use printf, – fpmurphy Mar 2 at 19:01
  • @fpmurphy, I do, of course. What I take issue with is that you're teaching other people to use echo, despite echo's own documentation indicating that it's best avoided. – Charles Duffy Mar 2 at 20:41
  • @CharlesDuffy There are millions of lines of shell code out there with echo statements and they work perfectly fine. The P in POSIX stands for Portable and anything in the APPLICATION USAGE section is generally focused on portability issues. If portability across multiple POSIX-compliant platforms and multiple shells, both old and modern, is required and echo is using -n as the first argument and escape sequences are used, then you have a case for not using echo. My example works OOTB in the two shells specified by the OP, i.e bash and 'dash` You are tilting at windmills! – fpmurphy Mar 3 at 4:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.