A piece of software wants to know for its configuration whether my man uses nroff or vt100 "catman" pages. How do I determine this?
Give nroff a go, most recent versions of 'man' use *roff by default.
Shows under Defaults that nroff is the default on my Ubuntu & OS X systems so should be a safe bet.
Short answer: just pick nroff.
Nroff is the traditional word processor for manual pages. “Catman” pages are usually manual pages that have been formatted by nroff for use on a text terminal. Catman pages have little formatting, just boldface and underline, and are justified for a particular terminal width (generally 80 columns). In the old days, it made sense to install preformatted (i.e. catman) pages to avoid the formatting time, but on a modern (or even not-so-modern) machine the time spent formatting is insignificant. Some systems have a cache so that each source page is formatted only once, but even that is falling into disuse. Having the source nroff pages has the advantage that they can be formatted for a different terminal width, for a different character encoding, or for viewing in a non-terminal setting (web browser, pdf, ...).
In fact, even if one of these is absent, most systems support both. For example, Linux systems rarely have any catman page installed but do support them. Conversely, the base OpenBSD installation comes with catman pages but supports nroff man pages (which are the norm in ports). A few systems support other formats as well, for example Solaris ships some pages in an SGML format.