After you run
exec &>filename, the standard output and standard error of the shell go to
filename. Standard input is file descriptor 0 by definition, and standard output is fd 1 and standard error is fd 2.
A file descriptor isn't either redirected or non-redirected: it always go somewhere (assuming that the process has this descriptor open). To redirect a file descriptor means to change where it goes. When you ran
exec &>filename, stdout and stderr were formerly connected to the terminal, and became connected to
There is always a way to refer to the current terminal:
/dev/tty. When a process opens this file, it always means the process's controlling terminal, whichever it is. So if you want to get back that shell's original stdout and stderr, you can do it because the file they were connected to is still around.