0

The following example is kind of classical way of using pipes. On right side of pipe we have a "simple" utility which reads data from stdin (pipe) and prints to stdout in reverse order:

{ cat <</EOF
Hello
World
/EOF
} | rev

Obviously the result is:

olleH
dlroW

Now I want to feed that data through pipe into a command/utility, which is part of a shell here-doc:

{ cat <</EOF
Hello
World
/EOF
} | ksh -x <</EOF
rev
/EOF

This does not work, no error message, nothing, piped data gets lost since ksh is redirecting stdin to here-doc.

I tried some kind of redirection:

set -x
{ cat <</EOF
Hello
World
/EOF
} | { exec 4<&0; ksh -x <</EOF; exec 0<&4 4<&-; }
rev <&4
/EOF

But this does not work either, it shows an error on rev command, obviously the redirection of stdin to fd4 is not passed through to subprocesses of ksh:

+ exec
+ 4<& 0
+ ksh -x
+ cat
+ 0<< \/EOF
rev <&4
/EOF
+ 0<< \/EOF
Hello
World
/EOF
+ rev
+ ksh[1]: 4: cannot open [Ungültiger Dateideskriptor]
+ exec
+ 0<& 4 4<& -

I'm stuck here.

My real application is much more complex: on left side of pipe I will do ssh to a source server, optionally sudo to a different user on that server and run tar to pipe some data files. On the right side of pipe I will have to ssh to a target server, optionally sudo to different user and finally run tar to receive the data from pipe.

Btw.: everything works fine if I do NOT make use of the here-doc on right side of pipe:

{ cat <</EOF
Hello
World
/EOF
} | ksh -xc "rev"

Even with ssh plus sudo plus tar this works fine. For some reason I would prefer to have a here-doc for the shell on the right side. As you can see ksh is my favourite shell within that application.

As requested: my original statement is like

ssh $SSHOPT $INST_FILE_USER@$INST_FILE_HOST "cd $IDIR && { tar -cvf - $IFIL && echo RCSRC0 >&2 || echo RCSRC1 >&2; }" 2>$TMPFILE.err.src | ssh $SSHOPT $TRG_USER@$TRG_HOST "cd $TDIR && { tar -xmvf - && echo RCTRG0 || echo RCTRG1; }" 1>$TMPFILE.out.trg 2>&1

It is working fine so far. As you can see, there is a minimum of error checking implemented, which allows for recognition of tar success. Now I must add sudo functionality, that means between the shell from ssh and cd command I must add some code to switch user. Is it really sudo, which will be used? Might be pbrun command since customers might prefer pbrun. These different scenarios can be dealt with much more flexible if here-docs are in place instead of 1 monster statement.

Update from May 21st: I tried to follow recommendations given so far, but I'm still stuck. Let's take a look at the answer given below, which is working fine for me:

echo yup | ksh -c 'exec 4<&0; ksh -c "rev <&5" 5<&4'
puy

If I start using ssh or using ssh combined with sudo my sample with redirection does not work any more:

echo yup | ssh admin@trg14 ksh -c 'exec 4<&0; ksh -c "rev <&5" 5<&4'
bash: 4: Bad file descriptor

echo yup | ssh admin@trg14 sudo -n -u trg4 ksh -c 'exec 4<&0; ksh -c "rev <&5" 5<&4'
bash: 4: Bad file descriptor

I have tried 2 scenarios with ssh and sudo combination: first one is using the here-doc as input to ksh:

Scenario 1:

echo yup | ssh admin@trg14 sudo -n -u trg4 -- ksh<</EOF
whoami && whoami && rev && whoami
/EOF
trg4
trg4
trg4

This is working so far. Note: all 4 commands are run as user trg4, rev does not get any input through pipe as expected due to the here-doc.

Therefore I tried to implement the solution suggested below: here-doc will be input as parameter to ksh:

Scenario 2:

echo yup | ssh admin@trg14 sudo -n -u trg4 -- ksh -c "$(cat<</EOF
whoami && whoami && rev && whoami
/EOF
)"
trg4
admin
puy
admin

The output from rev is as expected, but: whoami is showing us that in this version/implementation only the first command is run as sudo user, the remaining commands will be run as ssh login user! In case of rev no problem, but if you need to access or write to database files you will fail with permission issues.

Conclusion from these 2 samples: scenario 1 might be the right track to run all commands under the correct sudo user, but it will need a solution to the redirection of stdin in order to be able to use the pipe.

Any help appreciated!

  • This might be an XY problem. What is your original problem you want to get solved? What is the "some reason" why you need a here-doc on the right hand side of the pipe? You seem to have thought about a possible solution for your problem that now creates a new problem, so maybe there is a completely different solution. Please edit your question to add more background information. – Bodo May 7 at 8:36
  • 1
    So the basic problem is that you want 2 conflicting uses for stdin for your ksh on the right hand side of the pipeline, you want it to be both the source of commands to run and the source of the data. Your interactive shell has the same issue, when you type cat > /tmp/x and then type some more text, how does the shell know that this extra text is not more commands rather than input to the cat command? In your redirection example which fails, does it still fail if you change the RHS to use () instead of {} and move the rev <&4 inside the parenthesis? – icarus May 7 at 8:44
  • @Bodo: as written in my last paragraph, the original problem is: ssh user@src sudo -u newuser cd mydir && tar -cvpmf - | ssh user@trg sudo -u anotheruser cd trgdir && tar -xvf - Generally spoken this command sequence (again a simplified version) is working fine. What if you need some further commands before calling tar? That's one reason why I prefer the here-docs. On left side the here doc is working fine as long as I make sure, that only tar is writing to stdout, any other message must go to stderr. On right side I may write to stdout/stderr, but I'm stuck reading from pipe! – Elmar May 7 at 9:19
  • @icarus: same error message, I tried all variants of {} or () and moved the rev inside the parenthesis; there is no real conflict with usage of stdin, I attempt to let ksh read from here-doc, but the data, which resides in pipe, should be read from one of the commands within here-doc. – Elmar May 7 at 9:24
  • Do you have 1) /dev/fd/... in your operating system or 2) process substitution <(...) in your outer shell? – Michael Homer May 7 at 9:25
0

An easy workaround is to pass the here-doc script as a -c argument to the inner shell:

{ cat <</EOF
Hello
World
/EOF
} | ksh -c "$(<</EOF
rev
# you can freely use " inside this script
/EOF
)"

will give

olleH
dlroW

This also works in zsh. Changing the $(<< to $(cat<< should let it work in any shell.

You can also fiddle with redirections, provided that you do them at command level instead of with exec:

... | ksh 4<&0 <</EOF
rev <&4
/EOF

and in this case you could also swap back the fds inside the here-doc script with exec 5<&4 4<&0 <&5 5<&-, but all this is a worthless complication IMHO.

Script level redirections in ksh

The problem from your third example could be reduced to a simpler testcase:

$ echo yup | ksh -c 'exec 4<&0; ksh -c "rev <&4"'
ksh: 4: cannot open [Bad file descriptor]

But:

$ echo yup | dash -c 'exec 4<&0; ksh -c "rev <&4"'
puy

That's because ksh will set the close-on-exec flag (O_CLOEXEC) on any fd opened with script-level redirections (eg with exec 4</path/to or exec 4<&0). This is allowed by the standard:

If exec is specified without command or arguments, and any file descriptors with numbers greater than 2 are opened with associated redirection statements, it is unspecified whether those file descriptors remain open when the shell invokes another utility.

That could be worked around by doing an extra command-level redirection:

echo yup | ksh -c 'exec 4<&0; ksh -c "rev <&5" 5<&4'
puy

With ksh93 (but not with mksh older than R57[1]) simply re-duping a fd over itself works:

$ echo yup | ksh93 -c 'exec 4<&0; ksh -c "rev <&4" 4<&4'
puy

[1] released in March 2019, fixed in this commit.

  • Since my code is run on customer systems I do not have any chances to make a choice for preferred shells. ksh is available on more or less every *nix system and ksh gives enough freedom for complex operations. Thanks for your solution to pass the here-doc as -c argument, this looks promising and I will evaluate it. – Elmar May 7 at 10:23
  • @mosvy: I fixed that in mksh some time ago, so it also works there. • @Elmar: there are many versions of ksh around, mind you. As for the original question: this cannot work because a | b means “take stdout of a and make it stdin of b”, so when you redirect stdin of b manually, that overrides the pipe. – mirabilos May 7 at 19:53
  • @mirabilos great. what was the first version of mksh where that was fixed? – mosvy May 8 at 4:35
  • @mosvy eh, look at the changelog… apparently in R57 “Make dup-to-self with ksh-style fd≥3 closing work; from catern via IRC” – mirabilos May 8 at 20:15
0

[you should ask a different question about a different problem, instead of piling on this]

Runing ssh with [command] argument(s)

When you run ssh with [command] arguments, as in ssh user@host foo bar baz, what happens is that its arguments will be joined by spaces into a single argument which will be passed via the -c switch to the login shell of the user on the remote machine (if that shell's /bin/sh, /bin/sh -c 'foo bar baz' will be run).

Now, what happens in your example:

echo yup | ssh admin@trg14 ksh -c 'exec 4<&0; ksh -c "rev <&5" 5<&4'

is that the arguments to ssh will be joined into a string:

ksh -c exec 4<&0; ksh -c "rev <&5" 5<&4

and the shell on the remove machine will first run

ksh -c exec 4<&0

and then

ksh -c "rev <&5" 5<&4

as two separate commands, and of course fd 4 in the second does not point to anything, because it was only redirected in the first command.

  • many thanks for your answer. And apologize for "extending" my first question, I thought the overall picture is important. If I understand correctly the whole string will be joined together and will got to -c argument of login shell: this means my first "ksh -c" might not be necessary and the remaining string 'exec....5<&4' should do the trick when given to login shell I tried that. – Elmar May 21 at 14:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.