I need ideas to create a way to remotely do (any) commands through multiples hops. The main problem is that the number of intermediary hops is variable! So I need 1) to describe the "topology" of the servers, and then 2) have a function?/script that can execute some commands on all of them, using the intermediary hops as needed to reach the final destination.

exemple of topology: (but this varies too)

server A communicates with server B directly.
servers C and D are "behind" B.
server E is behing C
etc.... (many servers, many indirect ways to reach some of them, some have multiple ways)

Then from A, you can execute "some command" on E with:
    ssh foo@B ssh bar@C ssh baz@E "some command"
 or you can build intermediary tunnels connecting A to **C** and then:
    ssh -P 1234 baz@E "some command"

I have a few "main hop" that can reach a lot of servers directly

I want to be able to do a (potentially complex) command on all 5 servers, via a not-too-complex command, allowing me to (in a script) do something like

do_on_all_servers "for dir in /something* ; do cd \"\$dir\" && something using $thisLOCALvariable ; done"
# with an easy way to say "from X, ssh to specificuser@Y using this ssh command and this username" to describe each intermediary hops
# dir is a remote variable, thisLOCALvariable is at the initiating script level, and the amount of "\" needs to be adjusted if going through multiple hops...

Additional constraint: no GNU tools, no "nc", just plain (very) old ssh & awk & similar old tools, to have a portable way. Portability must be high (use of nifty ">1990?" options and tools must be avoided). And I would prefer not have to do this by copying a script over the final destination & ssh via multiple hops that script, but this can be ok if you think it's simpler/better.

Right now I'm using variables to concatenete ssh's .... but this doesn't help for the how-many-"\"-do-I-need problem.

My idea include: stringing variables containing a "shh user@host" together. Or maybe using this very neat https://stackoverflow.com/a/15198031/1841533 tunneling way, which if I can do it properly allow to do scp, etc using the proper local port to reach the final destination "directly" (but not sure that this way is available on all hops).

  • 2
    Did you check -W host:port option to ssh? It works fine with ProxyCommand in your ssh_config. If the topology is not changing in time, it should be quite easy to create a configuration file which would make a layer of abstraction to do your job. – Jakuje Jan 14 '16 at 11:41
  • Unless there's an existing solution out there, this feels potentially too broad a topic. It's quite a complex requirement, and it might be better for you to start developing a script and then query individual bits you can't get to work. Let's see what other people say though, I could be wrong. – EightBitTony Jan 14 '16 at 11:46
  • @Jakuje: thanks for the pointer, I'll get a look at this. Right now I'm playing with the (nifty) stackoverflow.com/a/15198031/1841533 way, as it seems quite portable and doesn't necessitate a change in config files... – Olivier Dulac Jan 14 '16 at 11:53
  • It is really ugly. I would rather change one local configuration file than set up different aliases in bashrc with a list of forwardings. – Jakuje Jan 14 '16 at 11:57
  • The question is not really broad but not-so-easy, and I hope it will attract some good hints on how to proceed (I had to resist the temptation to lure some shell gods such as @stephane-chazelas to read it, but I thought that using the @ notation would be both awkward (with regard to the general way of doing things on this site), and potentially unpolite... oh, oops, I just did... now where is the edit-comment button? ^^ – Olivier Dulac Jan 14 '16 at 11:58

You could write/use a shell-function wrapping ssh that generates a custom config-file which uses nested ProxyCommands to reach your final host. The function would then call ssh with -F pointing to the generated temporary config file.

Something like this:

sssh() {
  echo -n "$1" |
  awk '
    BEGIN {RS=" "}
    { split($0,a,":"); u=a[1]; p=a[3]; h=a[2]
      print "\nHost "h"\n\tUser "u"\n\tPort "p;
      if (hop) print "\tProxyCommand ssh -q -W %h:%p "hop;
      hop=h }
    END {print "\nHost *\n\tControlMaster auto\n\tControlPath /tmp/%r@%h:%p"}
  ' > $TMPF;   TMPA=($1); TMPAA=(${TMPA[-1]//:/ }); TMPH=${TMPAA[1]}; shift;
  ssh -F $TMPF $TMPH $@;
  rm $TMPF; }

When run like that:

sssh "user1:so.me.ho.st:1234 user2:router.internal:2345 user3:device.internal:3456" do_complex_command

would generate a tmp-file like this:

Host so.me.ho.st
    User user1
    Port 1234

Host router.internal
    User user2
    Port 2345
    ProxyCommand ssh -q -W %h:%p so.me.ho.st

Host device.internal
    User user3
    Port 3456
    ProxyCommand ssh -q -W %h:%p router.internal

Host *
    ControlMaster auto
    ControlPath /tmp/%r@%h:%p

and finally run:

ssh -F /tmp/tmp.IUVSRrer45 device.internal do_complex_command

This will execute do_complex_command on the last "inner-most" host. If you need your command execute on all intermediaries, you will have to adapt the function a little.

  • thanks for the input, but I can not edit those files everywhere I'd need to... I'm looking for the most portable possible solution, ie a single script that handles the hops by itself. I found solution (not pretty, but it works and is usable) so I'll answer here as well, but the question is still open in case someone gets an even better idea out... – Olivier Dulac Sep 22 '16 at 17:23
  • I changed it for my <4.3 version : TMPAA=${TMPA[@]: -1} ; TMPAA=(${TMPAA//:/ }); TMPH=${TMPAA[1]}; ... but when I run it with sssh "root:host1:22 root:hop2:22" "hostname;hostname" : it doesn't use 'root' to connect to host1, but my own username? (the "hostname ; hostname" part is to check that your way prevents something : ssh user@host1 user2@host2 "hostname ; hostname" : will display host2 and then host1 ! (ie the command was not run on last hop, but was splitted at the first ';' and the rest run on the first hop) – Olivier Dulac Jan 19 '17 at 13:52
  • ok, I changed also: hop=h to hop= u "@" h : now it works ! (with those 2 changes : the one above because I don't have a bash >=4.3, and that one to include the username in the hop) : Thanks a lot, I'll now acccept your answer ! – Olivier Dulac Jan 19 '17 at 15:42
  • thanks, great that it now works for you. If you want, you can also edit the answer to reflect your changes, @OlivierDulac – Alex Stragies Jan 19 '17 at 15:51
  • Thx, I'll update it but there is still something that does not work. Working: retrieving remote tar : I changed $* to be "$@", and now I can: sssh "user@host1:22 user2@host2:22" "cd /path/on/host2 && tar cf - *" : this works. But the reverse doesn't (stdin gets lost at some point) : { cd /local/path && tar cf - ;} | sssh "user@host1:22 user@host2:22" "cd /remote/path/on/host2 && tar xf - " : tar receives 0 bytes (see it by replacing tar ... with wc -c ) – Olivier Dulac Jan 19 '17 at 16:09

I found a quite portable solution (doesn't need any edits to ssh config files, and is easily contained withing the calling script itself, and is quite easy to use) : just string together the ssh's, and the last one calls the shell (ksh or bash work well for "a bit older" unixes) :

echo "for remotedir in /some/path/*/ ; do \
      cd \"\$remotedir\" && \
      something using $thisLOCALvariable \
      ; done" | ssh user1@host1 ssh user2@host2 ssh user3@host3 "bash"
# note: the 'obvious' approach: 
#     ssh u1@h1 ssh u2@h2 "cmd1 ; cmd2; ..."
#  does NOT work : it would execute "cmd2; ..." on h1, not on h2 !!!
#  only cmd1 would be executed on h2 ... BE CAREFUL ! (at least on a not-too-old cygwin)

That exemple shows we can mix local variables and remote ones... and the need to "escape" the remote ones, as well as remote double-quotes, etc... some care needs to be taken to ensure we do what we meant to do.

But it is quite handy, as there is always the same (only 1!) level of quoting, even if we string together 3 or more ssh nesting, which is quite handy... It's similar to what $(cmd) brings compared to the old ` (backtick) approach: it simplifies the writing without having to take into account the level of nesting.

caveat: having this as parameters to a script (like I asked in the question : on_all_servers "for dir in /something* ; do cd \"\$dir\" && something using $thisLOCALvariable ; done" ) is a tad bit more difficult to figure out, as the calling shell first interpret the "\" and quotes... so for now I use the command-line version (or a similar construct withing a script, but not a script that gets the commandline from its arguments)...

I still leave the question opened, hoping someone gets an even better solution (or an improvement of this one, ie : allow a similar command to be passed as parameter to a "calling script")

Edit: I also found out how to do remote tar extraction using this method! It is not trivial, as stdin needs to be given twich to the remote shell, once to get to the proper directory, then the local tar cf - needs to be forwarded to that remote tar in the proper location : here it goes:

# the easy one, nothing special needed: from remote to local:
echo "cd /remote/path && tar cf - *" | ssh user1@host1 ssh user2@host2 ssh user3@host3 "bash" | ( cd /local/path && tar xvf - )

# and the hard one: the other way around: needs a trick:
( cd /local/path && tar cf - * ) | { { echo "cd /remote/path/ && tar xvf -" ; cat ;} | ssh user1@host1 ssh user2@host2 ssh user3@host3 "bash" ;}

#once again: this trick allows you to have 1, 2, or any level of ssh.
# PLEASE NOTE: I just run bash, and feed it the command, to avoid
#     the command to be split between the remote hose and the first hop!
#  ex of the way the 'obvious' way fails:
#       ssh user1@host1 ssh user2@host2 "hostname ; hostname;"
#    will return not twice the name "host2", but "host2" then "host1" !! 
#    which means the first part of the command was executed on the last hop,
#    but then the rest of it was executed on the first hop!!! DANGEROUS!
#    that is why I choose instead to use that way:
#       echo "hostname; hostname" | ssh user1@host1 ssh user2@host2 "bash"
#    this one properly executes the full (complex) command on host2

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.