I had experience to work with framebuffer and found it pretty well to make own graphical applications. But the problem is that framebuffer is too slow.
1) Make a new
tty which will work for framebuffer only
2) Avoid framebuffer on other TTYs
3) Provide fast read-write for /dev/fb
*4) Use GPU for framebuffer operations