nano is small. In this case, it limits the choices to the 8 predefined ANSI colors (plus bright/bold) so that it can use the predefined symbols from curses.h (such as COLOR_BLUE) as a guide to naming.
Many terminals support 256 predefined colors; nano can't take advantage of them, but Vim can.
Terminals which allow directly specifying the R/G/B content of a color are an exception rather than a rule—unlike GUIs.
Some terminals (including Xterm, which I maintain) support the escape sequence \e]4;N;#RRGGBB\a to change palette color N to the specified RGB value, and \e[38;2;R;G;Bm to set the foreground color to the closest approximation in the palette of the specified RGB value (use 48 instead of 38 for the background color). However, changing a palette color is not useful for nano, because it is taking advantage of the existing palette, in contrast with (the much larger) Vim, which can do this with an add-on.
On writing the above in December 2015, the most recent release of nano was version 2.4.2 (July 2015). At that point, nano was 23336 lines (7657 statements) in C, which was a small fraction of the 131621 lines of text files (counting the ".po" message files). At the moment (October 2021, six years later), the program size is about the same size (fewer lines, more statements), but the other text files have roughly doubled the size of its source-tree (253036 lines). It's not exactly "small" any longer (but nowhere near the size of vim). A couple of weeks before releasing nano 5.0 in July 2020, the developer added eight names for entries in xterm's 256-color palette, in src/rcfile.c:
const char hues[COLORCOUNT][8] = { "red", "green", "blue",
"yellow", "cyan", "magenta",
"white", "black", "normal",
"pink", "purple", "mauve",
"lagoon", "mint", "lime",
"peach", "orange", "latte",
"grey", "gray" };
short indices[COLORCOUNT] = { COLOR_RED, COLOR_GREEN, COLOR_BLUE,
COLOR_YELLOW, COLOR_CYAN, COLOR_MAGENTA,
COLOR_WHITE, COLOR_BLACK, THE_DEFAULT,
204, 163, 134, 38, 48, 148, 215, 208, 137,
COLOR_BLACK + 8, COLOR_BLACK + 8 };
That doesn't appear to be extensible (but at least it uses ncurses). However, it does not address OP's question because it does not provide a hex or RGB method for configuring nano. In developing ncurses, I created an example which reads the X11 rgb.txt file, as part of making the program display X pixmap files in color. But it also reads and displays using a data file for xterm's 256-color palette. For screenshots, see the discussion of the picsmap program.
