sudoers man page:
An exclamation point (‘!’) can be used as a logical not operator in a list or
alias as well as in front of a Cmnd. This allows one to exclude certain values.
For the ‘!’ operator to be effective, there must be something for it to exclude.
For example, to match all users except for root one would use:
If the ALL, is omitted, as in:
it would explicitly deny root but not match any other users. This is different
from a true “negation” operator.
should be what you are looking for.