I'm exploring the notion of having a shell script input and output two dimensional structures of strings, with the limitation that the strings do not contain newlines. They can contain spaces.

The way that I want to try to do this is by using unescaped whitespace as a delimiter for the horizontal axis, and the newline is the delimiter for the vertical axis.

To experiment with this, I am using the following small tool to inspect arguments:

❯ cat argshow                     
while (( "$#" )); do              
  echo '$'"${ITERATION}: ${1}"    
  (( ITERATION ++ ))              


❯ argshow abc\ def zzz
$1: abc def           
$2: zzz               

With this convention that I'm establishing, here is my first test case:

echo 'abc\ def ghi\njkl'

This represents a value (presented in JSON) of

[["abc def", "ghi"], ["jkl"]]

OK. Proceeding onward.

I first tried this. I ran this in my zsh shell.

❯ echo 'abc\ def ghi\njkl' | while read -r line; do argshow $line; done 
$1: abc\ def ghi                                                        
$1: jkl                                                                 

I'm not very happy about that because argshow has received a single argument in the first line when it should be two. The use of -r for read was able to recover the backslash, which is nice (i'll worry about tabs later...).

I ran the same in my bash shell:

$ echo 'abc\ def ghi\njkl' | while read -r line; do argshow $line; done
$1: abc\
$2: def
$3: ghi\njkl
$ echo $'abc\ def ghi\njkl' | while read -r line; do argshow $line; done
$1: abc\
$2: def
$3: ghi
$1: jkl

This is where i've reached the limits of my bash knowledge. None of these are what I expect. Why is it that I must insert an eval in order to make this work like I want?:

$ echo $'abc\ def ghi\njkl' | while read -r line; do eval argshow $line; done
$1: abc def
$2: ghi
$1: jkl

I was under the impression that if I do not use any quotes when placing a bash variable that it would just get expanded and split to whatever it needs to be. After all, this is why shellcheck yells at me when I use shell variables in shell scripts without double-quoting them. I thought they would always get expanded out and so if the variable happened to have say 2 space chars in it, that it becomes 3 arguments instead of the expected 1 argument.

I cannot use this method to make scripts that I use on a regular basis. There are a lot of evil vulnerabilities involved when eval enters the script. It would be tempting because it allows for such free features as placing arguments into quotes, potentially being able to escape newlines, and it is easy to remember the syntax because of familiarity, but it would be far too dangerous to use in a script as it would be trivial to insert an unescaped semicolon to cause code execution.

Perhaps I've truly found the limit of shell scripting and clearly need to implement this with a real programming language. That is fair, but I'm hoping there's some simple option I could use to work around the issue and follow through on this hack.

Yes I also know I can probably do something like pass bash arrays around, but I'm hoping to be able to just dump these data streams out to simple files, that is the entire appeal of this, is that it has a super basic implicitly human readable format.

For more context on the task at hand... I am moving toward a more computer-assisted form of a data management scheme. The idea is that if I use something like a 512GB SD card in my camera/drone/etc. I'd like to be able to keep some older data remaining on the memory card, for convenience purposes, and also for workflows like sometimes I wanna dump some content onto my Macbook to view it more easily. When it comes to migrating this to ZFS pool at my workstation, the issue crops up of which files are already saved so I need not copy them back over and waste space. Sometimes if I fixate on that aspect too much, what happens is failure to pick up on some new stuff that should be backed up. It's always a time-consuming process to vet the remaining contents of a disk (that's probably nearly full) before erasing it to be able to continue to use it.

As such, the approach which I use now is periodically build a storage pool database find /pool /pool2 -type f -printf "%M %s %t %p\\n" > ~/Dropbox/find_zfs, then this database can be used pretty easily for different things. What led me to this question was in the course of writing the following script (I named it trawl):


# provide a list of dirs and for each of their immediate child file and folder names, trawl through 
# a standard file listing record (this is the FIRST arg) (the kind I ususally use with fzf) to 
# look for any matches. emit human readable output on stderr, and stdout will be a list paths.

# That output will be organized for now as a flat list. The idea here is that the list-of-lists 
# structure of it can trivially be recovered by a wrapping script that knows the input args and can 
# correlate them back out. Inefficient, but simple.

# TODO TODO TODO implement a sanitizer on the input and do a fuzzy check for that to further 
# eliminate manual checking via fzf and suggest matches when they are found


for DIR in "$@"; do
    >&2 echo "CHECKING DIR: $DIR"
    ls -1 "$DIR" | while read -r CHILD; do
        RG_OUT=$(rg -F "$CHILD" "$FIRSTARG")
        if [ $? -eq 0 ]; then
            >&2 echo "$DIR/$CHILD is found with $(echo "$RG_OUT" | wc -l) hits; skipping!"
            >&2 printf "NOT FOUND; adding: "
            echo "$DIR/$CHILD"

I also made a recursive version of to more exhaustively check every single file instead of just the first level children of a given directory.

So as you can see, the idea with this is that I can basically implement semi-automated file backup deduplication.

Of course, before someone mentions it, I could potentially just enable deduplication capability for ZFS itself, but it's widely accepted to be only worth the various performance costs it imposes on very narrow use-cases of which (primarily) bulk media backup is not one.

How does this fit in, you might ask, well I wanted a "sync point" which is what trawl implements here, the two axes of the two dimensional structure are: (X) all the individual child files in an input directory, and (Y) the separate input directories.

I did end up side-stepping the two dimensional requirement as you can see with my script's comment, as it turns out not to be such an important point and I can reverse it out after the fact. Indeed if I do want to save it, then it would be most sensible to emit JSON.

With this then, the workflow is now

  • plug in disk (if applicable)
  • trawl ~/Dropbox/find_zfs srcdir/ srcdir2/
  • visually review the output to confirm it makes sense
  • Run it again but piped into mkdir -p /pool/backup-$(date +%F); while read -r LINE; do mv "$LINE" /pool/backup-$(date +%F); done

It's a bit funny because I do have a bona fide typescript program I wrote to accomplish the same thing for me and it does have the same capability but it's just simply not as natural to implement an interactive and flexible workflow.

  • Why don't you use JSON as input and process that with some JSON-aware tool to parse out the arguments? That would additionally allow you to accept arguments with embedded newlines.
    – Kusalananda
    Aug 19, 2022 at 5:48
  • (cont.) My thought with the previous comment is to use any existing structured format rather than trying to invent a new one.
    – Kusalananda
    Aug 19, 2022 at 6:11
  • In a POSIX shell, argshow $line should be splitting $line on whitespace, since the variable isn't quoted. zsh is deviating from POSIX here. That's probably better from a reliability point of view, but it gives rise to a different shell language.
    – Kaz
    Aug 19, 2022 at 7:41
  • The need to insert eval indicates that you're processing shell syntax. If your read command is reading shell syntax from its input source, or a fragment of shell syntax, and you want to glue that shell syntax fragment onto a command to create a command line, then you use eval.
    – Kaz
    Aug 19, 2022 at 7:44
  • E.g. shell_syntax='$(basename "foo/bar"); eval "echo $shell_syntax" will glue together the text echo $(basename "foo/bar") and then evaluate the whole thing as shell syntax.
    – Kaz
    Aug 19, 2022 at 7:46

3 Answers 3


If you want several words in a variable, you use an array, not a scalar variable, if you want read to treat \ as an escaping operator (for the field separators and for line continuation), omit the -r, that's what read does by default without.

So, in ksh / zsh / yash:

read -A fields; showargs "${fields[@]}"

Or in bash:

read -a fields; showargs "${fields[@]}"

(beware read splits on $IFS characters, which happens to contain space, tab and newline (and nul in zsh) by default).

In zsh, you can have an unquoted scalar variable expansion split on $IFS characters like in other Bourne-like shells by using $=line instead of $line (mnemonic: $= looks like scissors). Or $=~line for the full split+glob that other shells do and that shellcheck warns you about. Or ${(s[ ])line} to split on space explicitly instead of what $IFS happens to currently contain. But here, that would not help you process backslashes.

To handle arbitrary shell-syntax quoting, not just backslash (to do shell syntax tokenisation and quote removal) without resorting to eval, see @thrig's fine answer about the z and Q parameter expansion flags of zsh. Though I'd add that if you want to preserve empty elements, you'd need:

$ line='foo\ bar "" blah'
$ fields=( "${(Q@)${(z)line}}" )
$ typeset fields
fields=( 'foo bar' '' blah )

Or Z[n] in place of z for newline to also be treated as a token separator as opposed to a command separator token.

In any case, that's doing full shell syntax (zsh syntax here) tokenisation, not limited to quote handling. For instance a a$(b c)<y>y<<<"a b" line would be tokenised into fields=( 'a$(b c)' '<' y '>' y '<<<' 'a b' ), not fields=( 'a$(b' 'c)<y>y<<<a b' ) as you might have wished.

For a shell with multidimensional arrays and data (de)serialisation¹ support, see ksh93:

$ printf '%s\n' $'( (\'a b\' a) (\'b\\c\n\' d) )' | 
> ksh -c 'read -C a; typeset -p a; printf "<%s>\n" "${a[1][0]}"'
typeset -a a=(('a b' a) ($'b\\c\n' d) )

Though last time I tested it (many years ago), it was pretty buggy, and might very well still be. I can't guarantee it will be much safer than using eval if the input is not guaranteed to be valid serialisation stream.

Oh! actually it's not:

$ printf '%s\n' '( [`uname>&2`]=1 )' | ksh -c 'read -C a'

Also beware that JSON strings can contain NULs but only zsh can have NULs in its variables. Using a proper programming language instead of a shell to process complex data structures would make more sense to me.

¹ For the record, the last beta version of ksh93 from AT&T released before the development team was disbanded there and on which ksh2020 (now abandoned) was based also had some experimental support for JSON (de)serialisation, though it was very buggy (and removed from ksh2020 as a result). It also has CSV input / output support.

  • Yes, thank you. From your and @QuartzCristal's answer I can get the strong sense that there are these subtle differences between handling commands themselves and handling expanded strings. And although it may be possible to learn about the intricacies and then reason about them, for practical purposes (especially considering maintainability) it's going to be folly, plain and simple. So, the premise of this question basically immediately qualifies it for reaching for a different tool.
    – Steven Lu
    Aug 19, 2022 at 22:22

read assigns all additional columns to the last name given, so all the columns end up in your line variable. In ZSH:

% echo 'abc\ def ghi' | while read -r line; do print -Rl $line; done
abc\ def ghi
% echo 'abc\ def ghi' | while read -r line and; do print -Rl $line $and; done
def ghi
% echo 'abc\ def ghi' | while read -r line and another; do print -Rl $line $and $another; done

So you actually would need to interpolate (eval, at worst) line to handle the backslashes and literal \n strings. In ZSH the (z) "split words as if zsh command line" is pretty close:

% echo 'abc\ def ghi' | while read -r line; do cols=(${(z)line}); printf ">%s<\n" $cols[@]; done
>abc\ def<

Also echo is not really portable, and probably should be replaced with printf where possible, or a shell specific builtin such as print in ZSH:

% printf "abc\ def ghi\njkl" | while read -r line; do cols=(${(z)line}); printf ">%s<\n" $cols[@]; done
>abc\ def<

Whoops, where did our jkl go? Silent data loss is no fun.

% printf "abc\ def ghi\njkl\n" | while read -r line; do cols=(${(z)line}); printf ">%s<\n" $cols[@]; done
>abc\ def<

There we go...but there's still that \ hanging around, maybe we can unquote it? Still in ZSH:

% printf "abc\ def ghi\njkl\n" | while read -r line; do cols=(${(z)line}); print -l ${(Q)cols}; done
abc def

Note that printf is supplying newlines; if literal \n are showing up then there are options to interpolate those as well, which is more work. In most other shells, you'll be doing an eval as they probably do not have suitable parameter flags to do what ZSH can. So that's either eval everywhere and hope nothing gets executed, or two different versions of the script, if you want to stick with using the shell for this project. And maybe some tests, I'm not 100% sure that (z) in ZSH will always do what you want for your input format.

(I do set a very low threshold for switching away from the shell, especially if tricky things like while are used, or if the script gets longer than about 20 lines.)

A Random Benchmark

The Perl is 1282% faster than the ZSH on a 265625 line input file derived from /etc/passwd.

#!/usr/bin/env perl
#our @array; # I'll let the shell folks puzzle out multi-dimensional arrays
while (readline) {
    #push @array, [ map { s/\\ / /gr } split /(?<!\\) / ];
    print join(' ', map { s/\\ / /gr } split /(?<!\\) /), "\n";

#!/usr/bin/env zsh
# do you like silent data loss? if so, remove the || check
while IFS= read line || [ -n "$line" ]; do
  fields=( "${(Q@)${(z)line}}" )
  typeset -p fields
  • map { s/\\ / /gr } split /(?<!\\) /) is not the correct way to extract space-separated fields while handling backslash escapes. That would split foo\\ bar into only one foo\ bar word instead of foo\ and bar for instance. . More like @fields = map { s/\\(.)/$1/gr } /(?:\\.|\H)+/g; . You'll also need some use open ':locale' to decode the input as text for a fairer comparison with zsh. Aug 19, 2022 at 15:35
  • I've used the only \ is special in other code (midnoi, under my lingua repository), which greatly simplifies the parsing, with the minor caveat that \ can never end a field. Perl is 1033% faster than ZSH with use open ':locale'; shells remain markedly inferior for this sort of task.
    – thrig
    Aug 19, 2022 at 15:48
  • That's not only about \ that can't end a field. When you use an escape character, it should be able to escape at least itself. In shells, \ escapes character except newline. foo\\\ bar baz should be split into foo\ bar and baz for instance. Aug 19, 2022 at 16:00
  • I disagree. A custom shell can do only what is necessary for the domain specific language in question, for example when \ is used nowhere else. Also, using a full lexer the Perl is 400% faster, and at that point would be trivial to expand to a much more rich grammer, if need be, and probably would still require a bunch of sleep statements to be competitive with ZSH. I guess you could leave those in for when management wants to see an improvement of some sort...
    – thrig
    Aug 19, 2022 at 17:02
  • I'm not arguing about whether a programming language is preferable over a shell, we're on the same page on that front, but about how to properly implement escaping. Aug 19, 2022 at 17:08

It happens to be that shells could be quite shallow when dealing with splitting (an unquoted variable expansion), or with read.

You introduce this issue when your code is do argshow $line

With bash:

$ set -- 1 2 3 4
$ echo ${@@A}
set -- '1' '2' '3' '4'

$ var='1 2 3 4'
$ set -- $var
$ echo ${@@A}
set -- '1' '2' '3' '4'         # that seems to work.

$ var='1 2\ 3 4'; set -- $var
$ echo ${@@A}
set -- '1' '2\' '3' '4'        # oops, why is that `\` ignored ? 

The \ is ignored because there is no rule to give a \ an special syntax (when expanded from a variable).

Yes, there is (still) an special rule about the \ when parsing a command line:

$ set -- 1 2\ 3 4
$ echo ${@@A}
set -- '1' '2 3' '4'

But that doesn't apply to variable expansions.

With read, we need separate lines, but that could be done either with printf or with the $'...' ANSI-C escape format.

$ printf '%s]n' a b c d

$ i=1; printf '%s\n' a b c d | while read -r line; do printf '%s %s \n' $((i++)) $line; done
1 a 
2 b 
3 c 
4 d 

$ var='a b c d'
$ i=1; printf '%s\n' $var | while read -r line; do printf '%s %s \n' $((i++)) $line; done
1 a
2 b
3 c
4 d           # it seems to work.

$ var='a b\ c d'
$ i=1; printf '%s\n' $var | while read -r line; do printf '%s %s \n' $((i++)) $line; done
1 a 
2 b\
3 c
4 d  

That is why there are many posts about: "quote variable expansions". And dealing with arrays is not the same as dealing with strings.

$ var=( a b\ c d)
$ printf '%s\n' "${var[@]}"
b c

# Or even
$ set -- "${var[@]}"
$ printf '%s\n' "$@"
b c

In short: a variable expansion is split on only IFS delimiters. Which generally reduce to one type of delimiter (whitespace).

$ ( IFS=''; var='a b\ c d'; printf '<%s> ' $var; echo )
<a b\ c d> 

$ ( IFS=' '; var='a b\ c d'; printf '<%s> ' $var; echo )
<a> <b\> <c> <d>

And that doesn't cover the effect of a quoted space (\ ) that works on a command line.

The zsh shell is even shallower, it simply doesn't split variable expansion (unless asked to).

% var='a b\ c d'
% printf '<%s>\n' $var
<a b\ c d>

% printf '<%s>\n' $=var

But the \ issue is still the same.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .