SMTP and ESMTP (the underlying protocols) that handle mail delivery have extensive RFCs (the original being RFC821, and more modern update RFC2821 and a Internet standards track protocol in RFC5321).
How mail servers deal with errors during delivery varies from mail server to mail server. Adding to the complication is the fact that a lot of them are configurable and easy to change the default behavior outlined in the RFCs.
The general rule of thumb, given the above caveats is:
Pick the highest preference MX record, or one at random if several records of the same preference exist (some times the random behavior is instead a round-robin algorithm). If the chosen host is "unreachable" (no route to host, connection refused or similar), try the next MX record of same preference or lower. As msw has mentioned, these are some what counter intuitive - the highest preference is 0 and records of a higher number are considered less preferential.
This is repeated until a connection is established, OR all hosts fail to respond, in which case the email is re-queued for later attempt at redelivery. Most mail servers will attempt this for a certain amount of time (usually something like 1 to 2 days), before it gives up and returns the email in a Non-Delivery Report (NDR).
If the connection is successful, the various steps of the RFC protocol dictate the general behavior of connecting MTAs. From the initial banner sent by the remote mail server, to each of the various command issued to it (from
RCPT TO and
DATA statements), the general rule of thumb is:
4xx transient error, try again later
With this code, the email is re-queued by the local mail server and delivery attempted at a later time (configured in the settings of that local mail server)
5xx fatal error, mail undeliverable
With this code, the email is considered undeliverable and the local sending mail server will (not always, but on most servers) generate an NDR (Non-delivery Report).
In terms of your question "If this server doesn't have the requested mail address", at
RCPT TO stage, most servers would respond with a
5xx code and your local mail server would generate an NDR.
Not all email servers are created equal
There are some caveats to this. MS Exchange for the longest time, would accept ALL emails regardless of incorrect recipients, unroutable domains and so forth, and then generate an NDR after the fact. Certain ISPs due to issues with spam and phenomenon known as Back Scatter, do not even generate NDRs and your mail "silently fail" (you never receive any notification of failed delivery).
You also have to take into consideration that the MTA (Mail Transport Agent, or mail server) is not always the end-point of delivery and MDAs (mail delivery agents - such as procmail) and MUAs (Mail User Agents) or "mail clients" such as thunderbird/outlook etc can be configured to "return" those emails with their own NDR-like responses. There are also such mechanisms as
.forward files which can get the MTA to redirect the email to another address after acceptance of the email. Certain mail servers (I know this is the case for Exim), will attempt to expand the
.forward at the point of the
RCPT TO stage of the SMTP conversation and if that expands to an unroutable address reply with the
5xx series of error codes mentioned above.
For a much more accurate and in-depth explanation read the RFCs mentioned above and the documentation of the MTA you are using (remembering that how it is configured may play a part in its behavior).