14

I have a program on a remote host, whose execution I need to automate. The command execute that program, on the same machine, looks something like this:

/path/to/program -a file1.txt -b file2.txt

In this case, file1.txt and file2.txt are used for entirely different things within the program, so I can't just cat them together. However, in my case, the file1.txt and file2.txt that I want to pass into the program exist only on my device, not on the host where I need to execute the program. I know that I can feed at least one file through SSH by passing it through stdin:

cat file1.txt | ssh host.name /path/to/program -a /dev/stdin -b file2.txt

but, since I'm not allowed to store files on the host, I need a way to get the file2.txt over there as well. I'm thinking it might be possible through abuse of environment variables and creative use of cat and sed together, but I don't know the tools well enough to understand how I would use them to accomplish this. Is it doable, and how?

  • 2
    catand sed are not the solution here. – Slyx Jun 3 at 15:55
  • Probably I'd be able to mount, but given proxies and security constraints I'm not sure I'd be able to get away with it. – Green Cloak Guy Jun 3 at 15:56
  • Do you have the permission in the remote machine to mount an ssh folder? – Slyx Jun 3 at 15:57
  • 1
    if you can open an ssh session from the remote machine to your local one, so there's no issue at the network level to mount an SSH folder. – Slyx Jun 3 at 16:00
  • Can you do forwardings? What systems and shells do you have at the local and the remote end? – mosvy Jun 3 at 17:46
17

If the files given as arguments to your program are text files, and you're able to control their content (you know a line which doesn't occur inside them), you can use multiple here-documents:

{
    echo "cat /dev/fd/3 3<<'EOT' /dev/fd/4 4<<'EOT' /dev/fd/5 5<<'EOT'"
    cat file1
    echo EOT
    cat file2
    echo EOT
    cat file3
    echo EOT
} | ssh user@host sh

Here cat is a sample command which takes filenames as arguments. It could be instead:

echo "/path/to/prog -a /dev/fd/3 3<<'EOT' -b /dev/fd/4 4<<'EOT'

Replace each EOT with something that doesn't occur in each of the files, respectively.

  • 1
    This approach is quite clever! If the files are not text, you could encode/decode them with something like base64 or even good ol' uuencode... Very nice! – filbranden Jun 3 at 22:07
  • 3
    Note that several (most) sh implementations implement here documents using temporary files, so it may not work for the OP if they are not allowed to store files on the remote host. – Stéphane Chazelas Jun 4 at 5:37
  • 1
    dash (the /bin/sh in debian, ubuntu, etc), busybox sh, the /bin/sh from FreeBSD and yash do not use temporary files for here-documents. I've actually tested it with a read-only chroot. Of course, the /dev/fd/ files have to be available -- on a stock FreeBSD system only /dev/fd/0-2 are, so fdescfs has to be mounted on /dev/fd. – mosvy Jun 4 at 10:56
  • 1
    I'm a fan of the ASCII information separators, which are designed to delimit fields without obstruction. U+001C is "Information Separator Four" (file separator, FS) and is ideal for this case, though binary files may make use of it coincidentally. Therefore, I'd suggest EOT="$(printf $'\x1c')" in advanced shells or else EOT="$(awk 'BEGIN{printf"%c",28}')" for compatibility. You'll need to quote it when you put it in place, such as echo "$EOT$EOT$EOT" (tripled to lower the odds of matching a binary file). – Adam Katz Jun 4 at 15:20
  • If you're going to use the tty why not use the tty command ? Unless I’m missing something - which I suppose is possible? – Pryftan Jun 4 at 15:57
14

Perhaps not exactly what you want... But maybe consider sending a tarball through the pipe opened by ssh?

You said that:

I'm not allowed to store files on the host.

It's possible that you don't have a writable home directory or other convenient location to store files long term, but I'd say it's unlikely you don't have any writable location, even if a temporary tmpfs that will be made available only for your particular connection.

Many programs (and even libc routines) require a writable /tmp, so it's very likely one will be available to you.

Then you could use a script that will unpack the tarball into a temporary directory, run your program and cleanup through the ssh connection.

Something like:

$ tar cf - file1.txt file2.txt |
  ssh host.name '
      set -e
      tmpdir=$(mktemp -d -t tmp.XXXXXXXXXX)
      cleanup () { rm -rf "$tmpdir"; }
      trap cleanup EXIT
      cd "$tmpdir"
      tar xf -
      /path/to/program -a file1.txt -b file2.txt
  '

This might need some extra care with file paths and there are some corner cases to consider (test for them), but the general approach should work.

If no writable directory is available, a possible approach would be to modify program to take a tarball as single input and unpack its contents to memory. For instance, if program is a Python script, then using the built-in tarfile module would easily accomplish something like that.

  • 3
    Even without going through the trouble of tarballing it - I've decided more or less to just write a file to /tmp/ and use it directly. – Green Cloak Guy Jun 3 at 21:48
  • 2
    The tarball approach has some advantages, as you use a single ssh connection (no need to test for success/failure) and multiple concurrent runs are not likely to run into each other (use and/or delete files in /tmp meant for separate runs.) But I think that's the best you'll get from ssh as it exists now. Good luck! – filbranden Jun 3 at 21:50
  • 3
    @GreenCloakGuy So after all, you can store files on the server just fine. Please modify the question accordingly. – mosvy Jun 3 at 22:04
  • 1
    @mosvy can, yes, but don't want to. The question stands, in a "is it possible to do this without storing any files", the answer to which appears to be "maybe, but not in my case" – Green Cloak Guy Jun 4 at 0:36
5

If you can set a TCP listener (can be a higher port), then you could use a second SSH session to establish the second input source with nc.

Example:

There's this script on the server (~/script.bash):

#!/usr/bin/env bash
cat "$1" | tr a 1
nc localhost "$2" | tr '[:lower:]' '[:upper:]'

And there are these two files locally:

$ cat a 
aaa
aaa
aaa
$ cat b
bbb
bbb
bbb

Now first start the second source ($serv is the server):

ssh "$serv" nc -l -p 2222 -q1 <b &

And run the command proper:

$ ssh "$serv" ./script.bash - 2222 <a
111
111
111
BBB
BBB
BBB
[1]+  Done                    ssh "$serv" nc -l -p 2222 -q1 < b
2

It was established in a comment that /tmp is writable, so simply copy over one of the files beforehand:

scp -p file2.txt host.name:/tmp/
ssh host.name "/path/to/program -a /dev/stdin -b /tmp/file2.txt && rm /tmp/file2.txt" < file1.txt

This also cleans up the copied file after a successful run (change the && to a ; if you want to remove it regardless of success, but then note that you'll lose the exit value).


If that's not acceptable, I'd propose tinkering with /path/to/program or a wrapper for it that can separate the two files from a single input stream such as:

awk 'FNR == 1 && NR > 1 { printf "%c%c%c", 28, 28, 28 } 1' file1.txt file2.txt \
  | ssh host.name /path/to/tweaked_program

This uses ASCII information separator four (file separator, FS) and has it tripled so that we minimize the chance of a binary file coincidentally containing that string. Your tweaked_program would then split the input given the separator and then operate on the two saved files as variables.

Of course, if you using a language that has libraries to deal with tarballs, a safer and cleaner approach would simply be to pipe tar into such code across ssh like this:

tar -zpc file1.txt file2.txt |ssh host.name /path/to/tweaked_program

And your tweaked_program would decompress and open the archive, save each file to a different variable, and then run the original program's logic on the variables.

1

If you're allowed to forward ports over ssh, and you have access to wget on the remote machine and to busybox on the local machine, you can do something like:

mkdir /tmp/test; cd /tmp/test
echo 1st_file > 1st_file
echo 2nd_file > 2nd_file

busybox httpd -f -p 127.0.0.1:11080 &
ssh USER@HOST -R 10080:127.0.0.1:11080 '
        cat <(wget -q -O- http://localhost:10080/1st_file) \
            <(wget -q -O- http://localhost:10080/2nd_file)
'
kill $!

(using cat as an example program which takes two file arguments).

Only the ability to forward ports via -R is essential -- instead of doing http, you can use other methods, eg. if your netcat supports the -d and -N options:

nc -Nl localhost 11001 < 1st_file &
nc -Nl localhost 11002 < 2nd_file &
ssh USER@HOST -R 10001:localhost:11001 -R 10002:localhost:11002 '
        cat <(nc -d localhost 10001) <(nc -d localhost 10002)

There may be ways to replace the <(...) process substitutions if the login shell on the remote machine isn't ksh- or bash- like.

All in all, this isn't too great -- better knowledge about the exact system/shells/config/permissions (which you haven't provided) could allow for smarter solutions.

-1

UPDATE: This doesn't really work, ssh has a very set idea of what stdin and stdout/stderr mean and doesn't really allow usurping stderr to read from it. I will delete this answer in a few days since it doesn't work. Thanks for the very interesting discussion!!!


Unfortunately there's not a great way to do this directly, since the ssh client will only pass the three file descriptors (stdin, stdout and stderr) to the server and it doesn't have provisions to pass additional file descriptors (which would be helpful for this particular use case.)

(Also note that the SSH protocol has provisions to pass additional file descriptors, only the ssh client doesn't implement a way to use that feature. Theoretically, extending the client with a patch would be enough to expose this feature.)

One hacky way to accomplish what you're looking for is to use file descriptor 2 (the stderr file descriptor) to pass the second file. Something like:

$ ssh remote.name \
      /path/to/program -a /dev/stdin -b /dev/stderr \
      <file1.txt 2<file2.txt

This should work, it just might cause an issue if program tries to write to stderr. You can work around that by re-juggling the file descriptors on the remote end before you run program. You can move file2.txt to file descriptor 3 and re-open stderr to mirror stdout:

$ ssh remote.name \
      /path/to/program -a /dev/stdin -b /dev/fd/3 \
      '3<&2' '2>&1' \
      <file1.txt 2<file2.txt
  • I haven't really tested this (typing on a phone) but at least in theory I'd expect it to work. Let me know if for some reason it doesn't. Cheers! – filbranden Jun 3 at 17:33
  • That won't work at all. Not even with a hacked ssh. AFAIK stderr doesn't use a separate channel, but some special kind of message. But yes, being able to access ssh channels as pipes or unnamed unix domain sockets (instead of having to do socket forwardings) would be great, it just doesn't seem to be possible in any kind or shape. – mosvy Jun 3 at 17:44
  • @mosvy, fd 0, 1 and 2 on the remote command will be 3 different pipes. The data travels through 3 different channels multiplexed in the ssh connection, but those are uni-directional (from client to sever for fd 0 and from server to client for 1 and 2), so even on systems where pipes are bidirectional, that won't work. On Linux, openening /dev/stderr in readonly mode gives on the other end of the pipe, so it would never work either. – Stéphane Chazelas Jun 3 at 17:56
  • Note that the last one assumes the login shell of the user on remote.name is Bourne-like (3<&2 is a Bourne shell operator, and sshd runs the login shell of the user to interpret the command sent by the client) – Stéphane Chazelas Jun 3 at 17:57
  • 2
    @StéphaneChazelas Nope, a single ssh channel is used for stdin/stdout/stderr, and the stderr data is transferred via SSH_MSG_CHANNEL_EXTENDED_DATA messages with the type set to SSH_EXTENDED_DATA_STDERR. You can read the whole thing here – mosvy Jun 3 at 19:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.