This syntax works for one remote source/destination:
rsync -avz -e "ssh -p $portNumber" user@remoteip:/path/to/files/ /local/path/
Which syntax should I use if both source and destination have different SSH ports?
If both of them have different ports, it is a good time to start thinking about using a configuration files, such as
Host server1 Port port1 Host server2 Port port2
and you are all set up.