imap.gmail.com
has many different IP addresses, which is unsurprising for a high-volume, high-availability service. The DNS server apparently returns two addresses (at least it does for me), but the set of addresses changes over time (it stays the same for a few minutes due to caching).
When you set the rule, the DNS server returns two addresses A1 and A2. The iptables program is smart enough to detect that and creates two rules, one for each of the two IP addresses.
A few minutes later, you run your email client, and its DNS requests returns two addresses A3 and A4, which most of the time won't be the same as A1 and A2. So mutt uses the first one A3, and that address is blocked by your firewall.
The firewall only sees IP packets. It cannot know what DNS name the email client uses. Indeed the email client could have used the IP address directly. IP packets are addressed to IP addresses, they don't contain host names. There might be host names inside the IMAP connection, but since you're using IMAPS, the connection is encrypted, and your firewall can't spy on the traffic.
The server sends its identity (in a certificate) during the SSL handshake, so you could inspect the traffic and abort the connection if the server's certificate is not to your liking. You can do that with iptables, though not 100% reliably (I think that only works if the string you're matching is inside a single TCP packet). To do it fully reliably, you'd need to set up a content-sensitive proxy.