4

I have two remote servers that I am trying to ssh and cat some files.. I want to input the output of the ssh to a awk command.

This is what I have got


ssh username@host1 “cat /tmp/test/*” ssh username@host2 “cat /tmp/test/*” | awk ‘ command ‘

But this is not working, if the host2 directory is empty then the output of the host1 is repeated twice.

What am I doing wrong here?

4
  • Aside from the issues we've addressed so far, I don't see why you'd get the host1 output twice just because host2 produces no output. We already know the command in your question isn't the command you're running though because those curly quotes would cause an error so maybe you're really doing something else beyond just using different quotes and, in any case, my best guess is something in your awk command which we can't see is causing that duplicate output.
    – Ed Morton
    Commented Dec 2, 2023 at 13:12
  • @EdMorton the OP is seeing repeated output because the command they are running is ssh host1 command shh host2 command which is interpreted as ssh host1 "command ssh host2 command", so the command run on the first host is cat /tmp/test/* ssh host2 cat /tmp/test/* so cat is given everything else as an argument. See my updated answer.
    – terdon
    Commented Dec 2, 2023 at 14:07
  • 1
    @terdon thanks, that makes sense but then the OP would be seeing some important error messages like cat: ssh: No such file or directory or similar that they decided not to tell us about which is.... disappointing. It also once again makes me think the code in the question isn't the actual code the OP is running.
    – Ed Morton
    Commented Dec 2, 2023 at 14:15
  • Yes, there are certainly error messages if this command is run. If this actual command is run, they would be bash: line 1: $'\342\200\234cat': command not found because of the fancy quotes, but if normal quotes are used, they will be as you said and I show in my answer. @NecroCoder for next time, please make sure to check the errors and include them in any future questions.
    – terdon
    Commented Dec 2, 2023 at 14:18

3 Answers 3

10

Beware of those curly quotes ( and ) some Windows text editors will use, use straight quotes instead (' or "). You should also be using 's, unless you have some reason to use "s, e.g. to let a variable expand, see https://mywiki.wooledge.org/Quotes.

You should be doing this if you want a single pipe to awk:

{ ssh username@host1 'cat /tmp/test/*'; ssh username@host2 'cat /tmp/test/*'; } | awk ' command '

but consider doing this instead so awk can distinguish which command the input came from in case you need that distinction in future:

awk ' command ' <(ssh username@host1 'cat /tmp/test/*') <(ssh username@host2 'cat /tmp/test/*')

Here's the difference:

$ seq 3 > file1
$ seq 2 > file2

Wrong output (because FILENAME always contains - and ARGV[1..2] are empty):

$ { cat file1; cat file2; } | awk '
    FILENAME == ARGV[1] { host="host1" }
    FILENAME == ARGV[2] { host="host2" }
    { print host, $0 }
'
 1
 2
 3
 1
 2

Wrong output (because FILENAME and ARGV[1] contain the same temp file descriptor but ARGV[2] is empty):

$ awk '
    FILENAME == ARGV[1] { host="host1" }
    FILENAME == ARGV[2] { host="host2" }
    { print host, $0 }
' <(cat file1; cat file2)
host1 1
host1 2
host1 3
host1 1
host1 2

Correct output:

$ awk '
    FILENAME == ARGV[1] { host="host1" }
    FILENAME == ARGV[2] { host="host2" }
    { print host, $0 }
' <(cat file1) <(cat file2)
host1 1
host1 2
host1 3
host2 1
host2 2

Other possible way to get the correct output with this approach of providing the input:

$ awk '{print host, $0}' host='host1' <(cat file1) host='host2' <(cat file2)
host1 1
host1 2
host1 3
host2 1
host2 2
7

You need to group the commands. The simplest way is to use curly braces which group them without creating a subshell:

{ ssh username@host1 'cat /tmp/test/*'; ssh username@host2 'cat /tmp/test/*'; } | 
   awk ' command '

You can also use parentheses, but this is slightly inelegant since it is creating a subshell where you don't need one. However, the end result is the same, and it really shouldn't make a difference in this context. Plus, you don't need to add the final ; as you do when using { }:

( ssh username@host1 'cat /tmp/test/*'; ssh username@host2 'cat /tmp/test/*' ) | 
   awk ' command '

The reason you are seeing duplicated output is because both cat commands are being run on the same host. Check the output when using localhost as the host for both commands, and after running set -x:

$ ls /tmp/test
file1

$ ssh localhost "cat /tmp/test/*" ssh localhost "cat /tmp/test/*"
+ ssh localhost 'cat /tmp/test/*' ssh localhost 'cat /tmp/test/*'
/tmp/test/file1
/tmp/test/file1
cat: ssh: Is a directory
cat: localhost: No such file or directory
cat: cat: No such file or directory

As you can see above, this is interpreted as "ssh localhost cat arg1 arg2 arg3 arg4", so the cat is run on localhost ( host1 in your case), and given as arguments /tmp/test/*, ssh, localhost (host2 in your case), cat and then /tmp/test/* again. So the contents of host1's /tmp/test are catted twice.

3

One possible way is to exec ssh command in subshell and pipe the result on awk:

(ssh username@host1 "cat /tmp/test/*"; ssh username@host2 "cat /tmp/test/*" ) | awk ‘ command ‘
1
  • mind the curly quotes around the awk command (and no need for double instead of single quotes around the ssh cat commands).
    – Ed Morton
    Commented Dec 3, 2023 at 10:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .