Unset setgid bit with chmod numeric mode

Question

I've just noticed some interesting behavior with chmod when unsetting the setgid bit:

$ mkdir test
$ chmod 2755 test
$ stat -c '%a %n' test
2755 test  # as expected
$ chmod 0755 test
$ stat -c '%a %n' test
2755 test  # what? see below
$ chmod 00755 test
$ stat -c '%a %n' test
755 test  # double what?!

Attempting to unset the setgid bit with chmod 0755 doesn't work, which is surprising. However, the man page indicates that this is the intended behavior:

SETUID AND SETGID BITS

chmod clears the set-group-ID bit of a regular file if the file's group ID does not match the user's effective group ID or one of the user's supplementary group IDs, unless the user has appropriate privileges. Additional restrictions may cause the set-user-ID and set-group-ID bits of MODE or RFILE to be ignored. This behavior depends on the policy and functionality of the underlying chmod system call. When in doubt, check the underlying system behavior.

chmod preserves a directory's set-user-ID and set-group-ID bits unless you explicitly specify otherwise. You can set or clear the bits with symbolic modes like u+s and g-s, and you can set (but not clear) the bits with a numeric mode.

(Emphasis added)

So it seems chmod 0755 isn't meant to unset the setgid bit. Why, then, however, does chmod 00755 unset it? chmod doesn't seem to have any use for five digits of a numeric mode. Again from the man page:

A numeric mode is from one to four octal digits (0-7), derived by adding up the bits with values 4, 2, and 1. Omitted digits are assumed to be leading zeroes.

(Emphasis added)

What's going on here? Why would chmod decide to ignore a single leading 0? Why does it not ignore two leading 0's?

(Debian Stretch 9.1, with chmod (GNU Coreutils) 8.6)

Answer 1

I've found it! This info is missing from the man page but is in the Coreutils manual online. To wit:

On most systems, if a directory’s set-group-ID bit is set, newly created subfiles inherit the same group as the directory, and newly created subdirectories inherit the set-group-ID bit of the parent directory. On a few systems, a directory’s set-user-ID bit has a similar effect on the ownership of new subfiles and the set-user-ID bits of new subdirectories. These mechanisms let users share files more easily, by lessening the need to use chmod or chown to share new files.

These convenience mechanisms rely on the set-user-ID and set-group-ID bits of directories. If commands like chmod and mkdir routinely cleared these bits on directories, the mechanisms would be less convenient and it would be harder to share files. Therefore, a command like chmod does not affect the set-user-ID or set-group-ID bits of a directory unless the user specifically mentions them in a symbolic mode, or uses an operator numeric mode such as ‘=755’, or sets them in a numeric mode, or clears them in a numeric mode that has five or more octal digits.

Reference: https://www.gnu.org/software/coreutils/manual/html_node/Directory-Setuid-and-Setgid.html

Answer 2

Maybe it's a bug in the chmod in Debian Stretch 9.1? It works as it should on Ubuntu 12.04, Ubuntu 16.04.3, CentOS 6.9 and CentOS 7.4.