The relevant concept here is attack surface.
Attack surface is the range of places where an attacker can look for, and potentially find, a vulnerability.
A directory that a compromised account can write to is attack surface that can be used to try to compromise other accounts, taking advantage of bugs in software those other accounts run and how that software parses the content of that directory.
Using shared cache directories requires that one trust the other users with write access to that directory to write content to that directory only in the expected form. If tools involved in rendering man pages have vulnerabilities, providing that trust can be foolish.
(Another form of attack on shared directories is creating symlinks there that point to a file whose contents an attacker would like to overwrite but doesn't have permissions to do so; when a different user account tries to update the cache entry associated with the file the symlink is at, a tool that wasn't carefully written can overwrite the target file instead of the symlink).
Note that attack surface mitigation should be done even when no open vulnerabilities in a piece of software are currently known, because the process reduces the chances of a future attack being successfully used against your system.