ORing with true in a command over ssh

Question

When I try to run pkill -f remotely via ssh, and try to discard the possible error code (to go on with the rest of my script even if no process is found), || true does not behave as I expect.

$ pkill asdf || true
$ echo $?
0
$ pkill -f asdf || true
$ echo $?
0
$ ssh pi@10.20.0.10 "pkill asdf || true"
$ echo $?
0
$ ssh pi@10.20.0.10 "pkill -f asdf || true"
255

I suppose that it is ssh that returns 255, not the command between quotes, but why?

Answer 1

Your supposition that it’s ssh itself that returns the 255 exit status is correct. The ssh man page states that:

ssh exits with the exit status of the remote command or with 255 if an error occurred.

If you were to simply run ssh pi@10.20.0.10 "pkill -f asdf", you’d most likely get an exit status of 1, corresponding to the pkill status for “No processes matched”.

The challenging part is to understand why an error occurs with SSH when you run

ssh pi@10.20.0.10 "pkill -f asdf || true"

---

SSH remote commands

The SSH server launches a shell to run remote command(s). Here’s an example of this in action:

$ ssh server "ps -elf | tail -5"
4 S root     35323  1024 12  80   0 - 43170 poll_s 12:01 ?        00:00:00 sshd: anthony [priv]
5 S anthony  35329 35323  0  80   0 - 43170 poll_s 12:01 ?        00:00:00 sshd: anthony@notty
0 S anthony  35330 35329  0  80   0 - 28283 do_wai 12:01 ?        00:00:00 bash -c ps -elf | tail -5
0 R anthony  35341 35330  0  80   0 - 40340 -      12:01 ?        00:00:00 ps -elf
0 S anthony  35342 35330  0  80   0 - 26985 pipe_w 12:01 ?        00:00:00 tail -5

Note that the default shell is bash and that the remote command is not a simple command but a pipeline, “a sequence of one or more commands separated by the control operator |”.

The Bash shell is clever enough to realise that if the command being passed to it by the -c option is a simple command, it can optimise by not actually forking a new process, i.e., it directly execs the simple command instead of going through the extra step of forking before it execs. Here’s an example of what happens when you run a remote simple command (ps -elf in this case):

$ ssh server "ps -elf" | tail -5
1 S root     34740     2  0  80   0 -     0 worker 11:49 ?        00:00:00 [kworker/0:1]
1 S root     34762     2  0  80   0 -     0 worker 11:50 ?        00:00:00 [kworker/0:3]
4 S root     34824  1024 31  80   0 - 43170 poll_s 11:51 ?        00:00:00 sshd: anthony [priv]
5 S anthony  34829 34824  0  80   0 - 43170 poll_s 11:51 ?        00:00:00 sshd: anthony@notty
0 R anthony  34830 34829  0  80   0 - 40340 -      11:51 ?        00:00:00 ps -elf

I’ve come across this behaviour before but I couldn’t find a better reference other than this AskUbuntu answer.

pkill behaviour

Since pkill -f asdf || true is not a simple command (it’s a command list), the above optimisation can not occur so when you run ssh pi@10.20.0.10 "pkill -f asdf || true", the sshd process forks and execs bash -c "pkill -f asdf || true".

As ctx’s answer points out, pkill won’t kill its own process. However, it will kill any other process whose command line matches the -f pattern. The bash -c command matches this pattern so it kills this process – its own parent (as it happens).

The SSH server then sees that the shell process it started in order to run the remote commands was killed unexpectedly so it reports an error to the SSH client.

Answer 2

Your remote command kills itself:

$ ssh 10.0.3.70 'pgrep -af asdf'
$ ssh 10.0.3.70 'pgrep -af asdf || true'
1018 bash -c pgrep -af asdf || true

pgrep and pkill will ignore their own process, but with the -f flag, they will find the parent shell:

$ pgrep -af asdf
$ pgrep -af asdf || true
$ bash -c 'pgrep -af asdf'
$ bash -c 'pgrep -af asdf || true'
9803 bash -c pgrep -af asdf || true

Answer 3

You ask pkill to kill anything that matched "asdf". You should tell it to match [a]sdf, that way it will still look for anything named "asdf", but won't see itself ( if you align asdf with [a]sdf, notice that the s is aligned with ] and not s.)

ssh 10.0.3.70 'pgrep -af "[a]sdf" || true'

It is a common trick also used with grep/egrep/awk/etc :

ps -ef | grep "something"  # will sometimes match itself too
ps -ef | grep "[s]omething" # will not match itself

# why it works:
# the commandline contains:     ps -ef | grep [s]omething
# and grep tries to find:                      something

This trick is old, and I saw it decades ago in the Unix faq (which is still a good read!)

To "automate" it, it is not easy, but usually each time you need to grep for a variable string regexp="something", you can try to do:

grep "$(echo "${regexp}" | LC_ALL='C' sed -e 's/[a-zA-Z0-9_-]/[&]/')" 
#  if regexp="something",  it does: grep "[s]omething"
#  if regexp="otherthing", it does: grep "[o]therthing"
#  if regexp="^thirdthing", it does: grep "^[t]hirdthing" #ok, kept the "^"
#BUT fails on : regexp="[abc]def", as it does: grep "[[a]bc]def" instead of grep "[abc][d]ef" ...