What shells support "value substitution" like MirBSD Korn Shell?

Question

On Android (which uses mksh, MirBSD Korn Shell), there's a special syntax of string substitution (called "value substitution"):

${|commands}

Instead of collecting output of the commands (as with `` and $()), the substitution result is taken from the $REPLY variable assigned inside the delimiters. It's special in that commands do NOT run in a subshell - they run in the same shell and has access to everything the current shell session has.

Debian has a mksh package for MirBSD Korn Shell, which acts exactly like that of Android.

What shells support a similar syntax, in that:

• Commands run in the same shell as the parent
• Substitution result is taken from other ways than file descriptors
  • It should be easy to note that reading outputs or pipes counts as FDs

Answer 1

That feature, called value substitution or valsub by Thorsten Glaser (aka @mirabilos), the maintainer or MirBSD and its shell mksh (derived from pdksh), is specific to mksh.

It was committed to the mksh code base on May 2nd 2013, and released in version R46 announced the next day on the mksh mailing list.

It's written on the back of the ${ body; } form of command substitution (called function substitution or funsub in mksh) copied from ksh93 in February that same year and also released in R46.

In ksh93, I/O from builtins is virtualised. Neither $(builin-cmd) nor ${ builtin-cmd; } involve any fork or I/O. So $(print foo) or ${ print foo; } expand to foo and meet your requirements of an operator that doesn't involve any fd.

In both forms, the print builtin doesn't write anything to any fd, but its would-be-output (trimmed of trailing newline characters) makes up the expansion. The difference between the two is that $(...) introduces a subshell environment (which contrary to other shells is not implemented by forking a child process), while ${ ...; } doesn't.

Now, to be able do that, ksh93 (a rewrite of ksh (itself from 1983) almost from scratch), all the I/O in the shell had to be rewritten specifically. When mksh added that ${ ...; } feature in 2013, it took a simpler approach which just records the output in a deleted temporary file, and reads that file's contents after the code in it returns to make up the expansion.

That means however that the output ends up stored on disk even if temporarily and the I/O means poorer performance than if the resulting data was just passed around in memory like in ksh93. So I suppose that's why Thorsten added that ${| ...; } separate form which can pass the value using a dedicated variable ($REPLY), and again doesn't require a major rework of the shell internals.

That means however that functions that are used in that fashion have to be written specifically to return their value in $REPLY (which can only be scalar not a list except via split+glob), and just becomes a bit of syntactic sugar. Example:

sanitize() {
  REPLY=${1//[!0123456789-]}
  local sign=
  case $REPLY in
    (-*) REPLY=${REPLY#-}; sign=-
  esac
  REPLY=$sign${REPLY//-}
}

print "$(( ${|sanitize "$1"} + ${|sanitize "$2"} ))"

Without it, you'd have to write:

sanitize "$1"; a=$REPLY
sanitize "$2"; b=$REPLY
print "$(( a + b ))"

One advantage over $(...) and ${ ...; } is that it doesn't strip trailing newline characters. For instance, $(basename -- "$file") is wrong as it doesn't work if $file ends in newline characters, while ${|basename -- "$file"} (assuming basename was rewritten as a function that returns the base name in $REPLY) wouldn't have the problem.

Other shells with constructs that can return values without involving I/O:

zsh

Someone did propose to implement a simplified version of mksh's valsub in 2019 which eventually evolved to this proposal, but as far as I know, it has not made it to zsh yet.

2024 edit mksh's ${|...} and ksh93's ${...;} have now been implemented in zsh and expected to be available in 6.0 or whatever the next release after 5.9 will be. There's also a ${|var|...} variant to allow returning the value in a variable other than $REPLY.

However, zsh has several alternative ways to have expansions being the result of arbitrary code without involving a subshell nor I/O.

Math functions

For arithmetic, zsh has the concept of math functions:

square() (( $1 * $1))
functions -M square 1

echo $(( square(5) + square(12) ))

That's limited to numbers though (integer or float) and can only be used in arithmetic expressions. The math functions themselves though can take non-numbers as arguments with functions -sM), so though very convoluted, you could do:

func() REPLY=foo$1; functions -sM func

echo ${$((func(bar)))+$REPLY} ${$((func(baz)))+$REPLY}

as an equivalent of mksh's:

func() REPLY=foo$1

echo "${|func bar}" "${|func baz}"

Dynamic named directories

zsh has another form of expansion than can be computed with shell code without I/O. That's using a customisation framework for tilde expansion called Dynamic named directories (see info zsh dynamic).

If you define:

autoload -Uz add-zsh-hook

valsub() {
  [[ $1 = n && $2 = '!'* ]] && eval "${2#?}" && reply=("$REPLY")
}
add-zsh-hook -Uz zsh_directory_name valsub

Then a tilde expansion of the form ~[!'REPLY=something'] would expand to something.

Tilde expansion is not done in every context, but you could also use that dynamic named directories feature as part of parameter expansion using that kind of trick described at that discussion about valsub support mentioned above.

e and + glob qualifiers

Globs can expand to the result of arbitrary code as well using the e (for evaluation) or + glob qualifiers.

Those are normally used to filter files based on the result of some code.

Like:

ls -ld -- *.txt(e['(( $#REPLY > 20 ))'])

To select txt filenames those length is greater than 20 characters. But can also be used to change the result of the expansion:

ls -ld -- *.txt(e['REPLY=$REPLY:r.html'])

(expand to txt files with the extension replaced with html). Or even:

ls -ld -- *.txt(e['reply+=($REPLY:r.html)'])

Returns both txt and html translation.

So you can actually do:

echo /(e['REPLY=foobar'])

For that to expand to the result of arbitrary code, here applying the qualifier to / which we know always exists. Or even a list:

printf '<%s>\n' /(e['reply=(foo bar)'])

The + qualifier is a variant that just takes a function name, so you can do echo /(+func) where func is the function that generates the expansion.

Again, like for ~ expansion, globbing is not done in every context.

es

es is a derivative of Byron Rakitzis's public domain clone of the Research Unix V10/Plan9 rc shell.

rc's functions can return a list of exit statuses (can be signal name or positive integers) and is made available to the caller in the $status list variable.

es extended it to be able to return any list of anything and instead of making it available in $status, the exit status (or function returned value) is obtained with the <={...} syntax.

So you can do:

fn foo { return foo$1 }
echo <={foo bar}

for instance.

Note however that only a returned value made of an empty list or a list whose elements are all empty or 0 is interpreted as being successful. So, for instance, here foo anything && echo bar would never output bar as foo always returns a value that is never interpreted as success.

ksh93

Beside $(...), ${ ...; } already discussed, there are a features that allow expansions to have dynamic contents without involving I/O:

disciplines

You can define a function that is invoked every time a variable is set or expanded. For associative array variables, those functions will have access to the subscript, so you could use that to pass one arbitrary argument to the function:

typeset -A valsub
function valsub.get {
  .sh.value=foo${.sh.subscript}
}
echo "${valsub[bar]}"

would output foobar.

math functions

ksh93 also has math functions, though with a different syntax from those of zsh:

function .sh.math.square x {((.sh.value = x*x))}
echo "$(( square(5) + square(12) ))"

Answer 2

To answer the question that is actually asked: None. No other shells have this.

As the MirBSD Korn shell changelog tells us, value substitutions were added by Thomas Goirand in 2014 to release 46.

No other shell has then or since copied this idea, to my knowledge. Some of them have the thing that value substitutions were derived from, but they do not actually have value substitutions.

Answer 3

The ${ cmds;} form of the command substitution of ksh93 runs cmds in the same shell, but otherwise captures the stdout like a regular command substitution. Example:

a=1; echo ${ a=2; echo wtf;}; echo $a
wtf
2

And the fact that it captures the stdout of the commands is exactly what makes it useful, since you don't have to save the output into some temporary file and then read it back, or set up a named pipe, or rewrite some hairy function to make it append its output to some variable instead of just writing it out.

And that's very much unlike that "value substitution" mksh feature, for which I'm unable to find any rationale. Why can't you assign the REPLY variable before, and then just use it as $REPLY?