Once you start X, the VT stops being handled as a "text device" and becomes a "graphical" one. In the olden days the distinction was clear: either the VT was relying on BIOS (at least to some extent), knew just a few text modes and was blazingly fast, or it was switched to a graphical mode, had more colours and/or larger resolution and was slower. These days the difference is not that clear (at least on Linux, can't tell about other UNIX variants), since the textual VT actually uses graphical mode, with the "translation" being done in the kernel. Nevertheless, you either let a text-based application open the VT or you leave it to X (or any other graphical front end like for example an implementation of the Wayland protocol).
As for the speed issues, choose your terminal right. Gnome terminal is likely to be slower than
xterm, which itself is way slower than for example
urxvt, unless you coerce it to use the same dirty tricks
urxvt does, by setting the appropriate X resource:
You very likely want to read Can a terminal emulator be as fast as TTY 1-6?, set up your terminal properly and use the graphical mode.
As a side note, some time ago (around 2007), I had problem with the nVidia framebuffer kernel driver, which was really slow on large resolutions like 1600x1200 - reading man pages in XTerm was much faster.