15

When I try to run pkill -f remotely via ssh, and try to discard the possible error code (to go on with the rest of my script even if no process is found), || true does not behave as I expect.

$ pkill asdf || true
$ echo $?
0
$ pkill -f asdf || true
$ echo $?
0
$ ssh pi@10.20.0.10 "pkill asdf || true"
$ echo $?
0
$ ssh pi@10.20.0.10 "pkill -f asdf || true"
255

I suppose that it is ssh that returns 255, not the command between quotes, but why?

29

Your supposition that it’s ssh itself that returns the 255 exit status is correct. The ssh man page states that:

ssh exits with the exit status of the remote command or with 255 if an error occurred.

If you were to simply run ssh pi@10.20.0.10 "pkill -f asdf", you’d most likely get an exit status of 1, corresponding to the pkill status for “No processes matched”.

The challenging part is to understand why an error occurs with SSH when you run

ssh pi@10.20.0.10 "pkill -f asdf || true"

SSH remote commands

The SSH server launches a shell to run remote command(s). Here’s an example of this in action:

$ ssh server "ps -elf | tail -5"
4 S root     35323  1024 12  80   0 - 43170 poll_s 12:01 ?        00:00:00 sshd: anthony [priv]
5 S anthony  35329 35323  0  80   0 - 43170 poll_s 12:01 ?        00:00:00 sshd: anthony@notty
0 S anthony  35330 35329  0  80   0 - 28283 do_wai 12:01 ?        00:00:00 bash -c ps -elf | tail -5
0 R anthony  35341 35330  0  80   0 - 40340 -      12:01 ?        00:00:00 ps -elf
0 S anthony  35342 35330  0  80   0 - 26985 pipe_w 12:01 ?        00:00:00 tail -5

Note that the default shell is bash and that the remote command is not a simple command but a pipeline, “a sequence of one or more commands separated by the control operator |”.

The Bash shell is clever enough to realise that if the command being passed to it by the -c option is a simple command, it can optimise by not actually forking a new process, i.e., it directly execs the simple command instead of going through the extra step of forking before it execs. Here’s an example of what happens when you run a remote simple command (ps -elf in this case):

$ ssh server "ps -elf" | tail -5
1 S root     34740     2  0  80   0 -     0 worker 11:49 ?        00:00:00 [kworker/0:1]
1 S root     34762     2  0  80   0 -     0 worker 11:50 ?        00:00:00 [kworker/0:3]
4 S root     34824  1024 31  80   0 - 43170 poll_s 11:51 ?        00:00:00 sshd: anthony [priv]
5 S anthony  34829 34824  0  80   0 - 43170 poll_s 11:51 ?        00:00:00 sshd: anthony@notty
0 R anthony  34830 34829  0  80   0 - 40340 -      11:51 ?        00:00:00 ps -elf

I’ve come across this behaviour before but I couldn’t find a better reference other than this AskUbuntu answer.

pkill behaviour

Since pkill -f asdf || true is not a simple command (it’s a command list), the above optimisation can not occur so when you run ssh pi@10.20.0.10 "pkill -f asdf || true", the sshd process forks and execs bash -c "pkill -f asdf || true".

As ctx’s answer points out, pkill won’t kill its own process. However, it will kill any other process whose command line matches the -f pattern. The bash -c command matches this pattern so it kills this process – its own parent (as it happens).

The SSH server then sees that the shell process it started in order to run the remote commands was killed unexpectedly so it reports an error to the SSH client.

  • 1
    While the answer correctly identifies the source a problem as in pkill kills its parent shell process because its arg list matches the regexp, I'll raise a terminology objection: x || y is not a compound command, it's a command list. – Stéphane Chazelas Nov 24 '17 at 20:15
  • @StéphaneChazelas Thanks for the feedback. I had been going by Bash's inclusion of conditional constructs as compound commands but I agree that it's more logically coherent to consider x||y as a command list. I’ve now edited my answer to include links to the various POSIX definitions. – Anthony Geoghegan Nov 25 '17 at 1:10
  • 1
    Note that in the general case, it's not so much that it can't optimise because it's a command list that it sill has another command to run (potentially). In zsh/ksh93/FreeBSD sh, false || pkill -f asdf would have pkill executed in the shell process. bash only does the optimisation when there's only one simple command. true; pkill -f asdf would also be a problem. – Stéphane Chazelas Nov 25 '17 at 8:18
9

Your remote command kills itself:

$ ssh 10.0.3.70 'pgrep -af asdf'
$ ssh 10.0.3.70 'pgrep -af asdf || true'
1018 bash -c pgrep -af asdf || true

pgrep and pkill will ignore their own process, but with the -f flag, they will find the parent shell:

$ pgrep -af asdf
$ pgrep -af asdf || true
$ bash -c 'pgrep -af asdf'
$ bash -c 'pgrep -af asdf || true'
9803 bash -c pgrep -af asdf || true
  • That makes sense! bash -c 'pgrep -af asdf' (without the || true) does not find itself. Why not? It has -f. – Gauthier Nov 24 '17 at 11:43
  • 2
    @Gauthier Actually, I think in this case, Bash is clever enough to realise that the command is a simple one (not a compound command) so it optimises by not actually forking a new process. I remember coming across similar behaviour before (I must update my answer). – Anthony Geoghegan Nov 24 '17 at 12:51
3

You ask pkill to kill anything that matched "asdf". You should tell it to match [a]sdf, that way it will still look for anything named "asdf", but won't see itself ( if you align asdf with [a]sdf, notice that the s is aligned with ] and not s.)

ssh 10.0.3.70 'pgrep -af "[a]sdf" || true'

It is a common trick also used with grep/egrep/awk/etc :

ps -ef | grep "something"  # will sometimes match itself too
ps -ef | grep "[s]omething" # will not match itself

# why it works:
# the commandline contains:     ps -ef | grep [s]omething
# and grep tries to find:                      something

This trick is old, and I saw it decades ago in the Unix faq (which is still a good read!)

To "automate" it, it is not easy, but usually each time you need to grep for a variable string regexp="something", you can try to do:

grep "$(echo "${regexp}" | LC_ALL='C' sed -e 's/[a-zA-Z0-9_-]/[&]/')" 
#  if regexp="something",  it does: grep "[s]omething"
#  if regexp="otherthing", it does: grep "[o]therthing"
#  if regexp="^thirdthing", it does: grep "^[t]hirdthing" #ok, kept the "^"
#BUT fails on : regexp="[abc]def", as it does: grep "[[a]bc]def" instead of grep "[abc][d]ef" ...
  • note: I'm aware that my 'fail' grep exemple, one could have kept the regexp as is, as it already won't match itself (the a, b or c will not match with the ']' of the commandline). But it is not trivial coming up with a test of the regexp. In general, the trick works. the one that automates will work most of the time. When not, some clever hacks (or manual intervention) will be needed. – Olivier Dulac Nov 24 '17 at 19:11
  • Also, (abc)?(def)? will have to be ([a]bc)?([d]ef)?... You can't parse regex with regex?! >:-) – wizzwizz4 Nov 25 '17 at 19:03
  • @wizzwizz4 I know. but your exemple already won't match itself. this is a complex thing, I just provided a simple solution for simpler cases – Olivier Dulac Nov 25 '17 at 19:05
  • @wizzwizz4 I already say so in my first comment... – Olivier Dulac Nov 25 '17 at 19:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.