I've changed my login shell to zsh from bash on an installation of Debian Squeeze with XFCE. When I launch xfce4-terminal, zsh starts but does not load my configuration.
You changed the shell in your user profile, but since you were already logged in your environment variables had already been set and xfce4-terminal was assuming you would want to use whatever was set as your shell when the environment got initiated. If you had logged out of everything on the machine or authenticated another way like logging in with SSH, it would have worked as well.
Many terminal programs have their own preferences for what shell to launch in them. These preferences can use any program you are allowed to run as the shell, including any system authorized shells. When you authenticate against the system, the first thing it will give you is always the shell in your profile, but after that you are free to run whatever.