No, it's not normal — it should take nowhere near a minute or two to get a response from the directory server. An extra second or two, sure, but not a minute or two.
I'd guess the most likely reason for adding an extra minute is SSSD is trying to contact a directory server that is down or no longer exists. If you're using DNS SRV records, make sure you don't have any old ones (pointing to decommissioned servers) around.
If you increase the log level in SSSD, it should log the particular timeouts its experiencing to help you find the responsible server.
And yes, when the cache expires, SSSD is going to have to query the directory server again — which if that takes minutes is going to be painful.