What changed is that
/bin/sh either became
bash or stayed
dash which got an additional flag
-p mimicking bash's behaviour.
Bash requires the
-p flag to not drop setuid privilege as explained in its man page:
If the shell is started with the effective user (group) id not equal to the real user
(group) id, and the -p option is not supplied, no startup files are read, shell functions
are not inherited from the environment, the SHELLOPTS, BASHOPTS, CDPATH, and GLOBIGNORE
variables, if they appear in the environment, are ignored, and the effective user id is
set to the real user id. If the -p option is supplied at invocation, the startup behavior
is the same, but the effective user id is not reset.
dash didn't care about this and allowed setuid execution (by doing nothing to prevent it). But Ubuntu 16.04's
dash's manpage has an additional option described, similar to
Do not attempt to reset effective uid if it does not match uid. This is not set by default to help avoid incorrect usage by
setuid root programs via system(3) or popen(3).
This option didn't exist in upstream (which might not be have been reactive to a proposed patch*) nor Debian 9 but is present in Debian buster which got the patch since 2018.
NOTE: as explained by Stéphane Chazelas, it's too late to invoke
"/bin/sh -p" in
system() runs anything given through
/bin/sh and so the setuid is already dropped. derobert's answer explains how to handle this, in the code before
* more details on history here and there.