Weird aspects of Unix usually exist for good reason, so you're right to look for one. In this case, though, the good reason has long since become obsolete, and you're looking at an antique artifact of a bygone era.
Just about the only "terminal" in existence today is xterm & variants. Their capabilities vary very slightly, in ways that matter to only a few programs. If you just use xterm, and never touch the TERM variable or peek at the terminfo database, your life will generally be better.
The TERM variable communicates information about the terminal to the application through the environment, cf. man xterm. Changing it doesn't change the terminal; it just represents different terminal functionality to the application.
In the days of hardwired terminals, it was necessary to set TERM to represent the attached terminal. In the case of xterm, the software can set the variable itself. A quick tour of the vim docs shows (as you mention in your comment) that you have to change it to support color. That's progress for you.
why today ... emulate these old terminals like VT and not have something new?
The answer is as much anthropological as technical.
Before the GUI, access to Unix machines was via dumb terminals e.g. VT-100. Shells and utilities like top already existed. When the GUI became technologically practical (in which X played a role) in the 1980s, Unix users still needed to use those programs, so xterm was invented to emulate ye olde VT-100.
It was meant as a stopgap. "Everyone knew" that terminals were the past and GUIs were the future, and everyone expected "everything" to be accessed via the GUI. The original Macintosh, for example, had no arrow keys because why would you need them? Surely the cryptic Unix command line, with its missing vowels and helpless help
$ help
help: not found
would soon go the way of drum memory and punch cards. And that did come to pass, in a way: 9 users in 10 running Windows or OS X never see the command line except when tech support drops by to fix something.
Then two things happened to the Unix GUI, such as it was. Windows in particular drained the money out of the market. There was a big move to standardize it (cf. Sun News and OSF Motif), and then it ground to a halt around 1990. Just about that time the Internet took off, and things graphical in Unix moved into the web browser. The motivation and the money (pretty much the same thing) to engineer a complete GUI for Unix and render everything in section 8 of the manual obsolete disappeared.
There is another reason, too, that very few foresaw: the command line has certain advantages over the GUI. Pipelines and regular expressions are remarkably powerful, not to mention repeatable with shell history and scripts. Even in the context of a GUI, the command line remained useful. So much so that it continues to be enhanced even today.
As your question suggests, what's needed is a re-examination of the assumption that the GUI would triumph, and a re-invention of the terminal as an integral part of it. We need a new terminal, with proportional fonts and bit-addressable graphics in the terminal.
Unfortunately, no one seems ready to do that. No corporate entity will undertake it; the market is huge, but still only a tiny proportion of computer users. The logical funder would be a government agency like DARPA, but human-interface research is considered "done" these days (didn't we already invent the GUI?). Until more people -- lots more people -- recognize the need, xterm is your friend, and likely to be your grandson's friend, too.
screenortmux)....bashrcare at best masking an underlying issue and only likely to cause further problems.to support pretty much any terminal there might be..But why? Why would we want to emulate a terminal from the 70s or 80s with no graphics or a green screen?