How to feed data through pipe into utility/command as part of shell here-doc?

Question

The following example is kind of classical way of using pipes. On right side of pipe we have a "simple" utility which reads data from stdin (pipe) and prints to stdout in reverse order:

{ cat <</EOF
Hello
World
/EOF
} | rev

Obviously the result is:

olleH
dlroW

Now I want to feed that data through pipe into a command/utility, which is part of a shell here-doc:

{ cat <</EOF
Hello
World
/EOF
} | ksh -x <</EOF
rev
/EOF

This does not work, no error message, nothing, piped data gets lost since ksh is redirecting stdin to here-doc.

I tried some kind of redirection:

set -x
{ cat <</EOF
Hello
World
/EOF
} | { exec 4<&0; ksh -x <</EOF; exec 0<&4 4<&-; }
rev <&4
/EOF

But this does not work either, it shows an error on rev command, obviously the redirection of stdin to fd4 is not passed through to subprocesses of ksh:

+ exec
+ 4<& 0
+ ksh -x
+ cat
+ 0<< \/EOF
rev <&4
/EOF
+ 0<< \/EOF
Hello
World
/EOF
+ rev
+ ksh[1]: 4: cannot open [Ungültiger Dateideskriptor]
+ exec
+ 0<& 4 4<& -

I'm stuck here.

My real application is much more complex: on left side of pipe I will do ssh to a source server, optionally sudo to a different user on that server and run tar to pipe some data files. On the right side of pipe I will have to ssh to a target server, optionally sudo to different user and finally run tar to receive the data from pipe.

Btw.: everything works fine if I do NOT make use of the here-doc on right side of pipe:

{ cat <</EOF
Hello
World
/EOF
} | ksh -xc "rev"

Even with ssh plus sudo plus tar this works fine. For some reason I would prefer to have a here-doc for the shell on the right side. As you can see ksh is my favourite shell within that application.

As requested: my original statement is like

ssh $SSHOPT $INST_FILE_USER@$INST_FILE_HOST "cd $IDIR && { tar -cvf - $IFIL && echo RCSRC0 >&2 || echo RCSRC1 >&2; }" 2>$TMPFILE.err.src | ssh $SSHOPT $TRG_USER@$TRG_HOST "cd $TDIR && { tar -xmvf - && echo RCTRG0 || echo RCTRG1; }" 1>$TMPFILE.out.trg 2>&1

It is working fine so far. As you can see, there is a minimum of error checking implemented, which allows for recognition of tar success. Now I must add sudo functionality, that means between the shell from ssh and cd command I must add some code to switch user. Is it really sudo, which will be used? Might be pbrun command since customers might prefer pbrun. These different scenarios can be dealt with much more flexible if here-docs are in place instead of 1 monster statement.

Update from May 21st: I tried to follow recommendations given so far, but I'm still stuck. Let's take a look at the answer given below, which is working fine for me:

echo yup | ksh -c 'exec 4<&0; ksh -c "rev <&5" 5<&4'
puy

If I start using ssh or using ssh combined with sudo my sample with redirection does not work any more:

echo yup | ssh admin@trg14 ksh -c 'exec 4<&0; ksh -c "rev <&5" 5<&4'
bash: 4: Bad file descriptor

echo yup | ssh admin@trg14 sudo -n -u trg4 ksh -c 'exec 4<&0; ksh -c "rev <&5" 5<&4'
bash: 4: Bad file descriptor

I have tried 2 scenarios with ssh and sudo combination: first one is using the here-doc as input to ksh:

Scenario 1:

echo yup | ssh admin@trg14 sudo -n -u trg4 -- ksh<</EOF
whoami && whoami && rev && whoami
/EOF
trg4
trg4
trg4

This is working so far. Note: all 4 commands are run as user trg4, rev does not get any input through pipe as expected due to the here-doc.

Therefore I tried to implement the solution suggested below: here-doc will be input as parameter to ksh:

Scenario 2:

echo yup | ssh admin@trg14 sudo -n -u trg4 -- ksh -c "$(cat<</EOF
whoami && whoami && rev && whoami
/EOF
)"
trg4
admin
puy
admin

The output from rev is as expected, but: whoami is showing us that in this version/implementation only the first command is run as sudo user, the remaining commands will be run as ssh login user! In case of rev no problem, but if you need to access or write to database files you will fail with permission issues.

Conclusion from these 2 samples: scenario 1 might be the right track to run all commands under the correct sudo user, but it will need a solution to the redirection of stdin in order to be able to use the pipe.

Any help appreciated!

Answer 1

An easy workaround is to pass the here-doc script as a -c argument to the inner shell:

{ cat <</EOF
Hello
World
/EOF
} | ksh -c "$(<</EOF
rev
# you can freely use " inside this script
/EOF
)"

will give

olleH
dlroW

This also works in zsh. Changing the $(<< to $(cat<< should let it work in any shell.

You can also fiddle with redirections, provided that you do them at command level instead of with exec:

... | ksh 4<&0 <</EOF
rev <&4
/EOF

and in this case you could also swap back the fds inside the here-doc script with exec 5<&4 4<&0 <&5 5<&-, but all this is a worthless complication IMHO.

Script level redirections in ksh

The problem from your third example could be reduced to a simpler testcase:

$ echo yup | ksh -c 'exec 4<&0; ksh -c "rev <&4"'
ksh: 4: cannot open [Bad file descriptor]

But:

$ echo yup | dash -c 'exec 4<&0; ksh -c "rev <&4"'
puy

That's because ksh will set the close-on-exec flag (O_CLOEXEC) on any fd opened with script-level redirections (eg with exec 4</path/to or exec 4<&0). This is allowed by the standard:

If exec is specified without command or arguments, and any file descriptors with numbers greater than 2 are opened with associated redirection statements, it is unspecified whether those file descriptors remain open when the shell invokes another utility.

That could be worked around by doing an extra command-level redirection:

echo yup | ksh -c 'exec 4<&0; ksh -c "rev <&5" 5<&4'
puy

With ksh93 (but not with mksh older than R57[1]) simply re-duping a fd over itself works:

$ echo yup | ksh93 -c 'exec 4<&0; ksh -c "rev <&4" 4<&4'
puy

[1] released in March 2019, fixed in this commit.

Answer 2

[you should ask a different question about a different problem, instead of piling on this]

Runing ssh with [command] argument(s)

When you run ssh with [command] arguments, as in ssh user@host foo bar baz, what happens is that its arguments will be joined by spaces into a single argument which will be passed via the -c switch to the login shell of the user on the remote machine (if that shell's /bin/sh, /bin/sh -c 'foo bar baz' will be run).

Now, what happens in your example:

echo yup | ssh admin@trg14 ksh -c 'exec 4<&0; ksh -c "rev <&5" 5<&4'

is that the arguments to ssh will be joined into a string:

ksh -c exec 4<&0; ksh -c "rev <&5" 5<&4

and the shell on the remove machine will first run

ksh -c exec 4<&0

and then

ksh -c "rev <&5" 5<&4

as two separate commands, and of course fd 4 in the second does not point to anything, because it was only redirected in the first command.