The first is a clumsy hack, the second is a common mistake.
These two tests do something completely different, which happens to produce the same outcome.
if test ! $(which brew); then
This tests whether the output of
which brew is empty.
brew is on the search path, then
which brew produces a word of output, so the
test command receives two arguments:
! and the path to
test has two arguments, the first of which is
!, it returns true if the second argument is empty (which isn't the case here) and false otherwise.
brew isn't on the search path, then the output of
which brew is empty, so the
test command receives a single argument which is
test returns true.
Note that this command will produce an error message and a failure status if the path to
brew contains whitespace, because that's what an unquoted command substitution means. It so happens that a failure status was the desired outcome here, so this works in a roundabout way.
This command does not test the exit code of
which brew. The exit code is ignored.
if ! which brew > /dev/null; then
This is the straightforward way to test if
which brew succeeds. It doesn't depend on anything brittle except
which brew is always called in both cases. Why would it matter that the output is redirected to
/dev/null? “Hide this command's output” does not mean “don't run this command”.
The proper way to test if
brew is not available on the command search path is
if ! type brew >/dev/null 2>/dev/null; then
See Why not use "which"? What to use then?