I'm trying to block a MAC address for a specific port.
iptables -I FORWARD -p tcp -m mac --mac-source 40:00:00:74:0A:00 --destination-port 80,443 -j DROP
A MAC is at the physical layer 2, and TCP and a TCP port are concepts from layers 3 and 4.
The rule won't work, either you block a MAC or don't.
Otherwise, to block a specific port, you need to block using an IP address + the TCP/UDP port.
To find the related IP address to the wanted MAC address, either you do it one time manually, or if doing it automatically over the long term, you need some process listening to ARPs in the network and/or to DHCP logs, and need to create on the fly rules to block the offending IP address that is using that MAC address.