As far as I know, PAM doesn't determine the user's shell, this is left to the application. PAM's session modules perform generic actions and checks that must be done for on every login using that particular service. If the application then wants to start a shell, it is free to do so, and will typically look up the shell in the user database.
Assuming your question is about OpenSSH, that's exactly what it does: once the user is authenticated, and the PAM session stuff has been done (if configured to use PAM¹), the ssh server looks up the shell in the user database (directly, not through the PAM library).
The user database isn't limited to
/usr/passwd and friends. On Linux (which I assume you're using since you mention
shadow), what makes up the user database is determined by the
passwd setting in
/etc/nsswitch.conf. In multi-computer setups, common additions to the local database are NIS and LDAP. If you want to use a shell that isn't the one in
/etc/passwd, this may be what to configure (although it would be a bit strange, and maybe people can offer better suggestions if you tell us what you're trying to accomplish).
If you want to have users without full shell access, the natural solution is to change
/etc/passwd to put a restricted shell — perhaps rssh to allow only a few file-copying-type applications such as scp, rsync and cvs. You can also use forced commands in the user's
If you'd like to see a trace of what the ssh server is doing, start the daemon as
ssh -ddd. You can also get the client's view with
ssh -vvv, though here the server's view is what will interest you most.
OpenSSH only uses PAM if it is configured with PAM support and the
UsePAM directive is set to
sshd_config. Even when it uses PAM, it offers other authentication methods in addition to PAM; in particular public key authentication does not go through PAM.