Why can't I know whether sending a mail fails?

Question

I'm implementing a CRM where I need to send mail to my customers. Of course, I found the problem that many seem to find: you don't know whether the mail was successfully sent. However, I don't seem to understand exactly why it's that way.

First, I understand that testing for an email can only be done by actually talking to the server in control of the email (using MX records and then the MAIL FROM and RCPT TO commands), and even then the server may refuse to answer those requests. I think I have that clear, and for example, that's how it seems that sites like http://verify-email.org/ work.

Then, I read from this comment to an answer here, that .NET's SmtpClient does the same thing to verify that the message can be sent, and if it fails it throws a SmtpFailedRecipientsException. So I seem to get it...

But then, I decide to try sending a mail to an invalid address: asdf@asdf.asdf (I hope .asdf is not one of those recently approved new TLDs). I use it on http://verify-email.org/ and it says it's invalid (because there's not even an MX record, obviously), but when I use SmtpClient it succeeds! Why?

Then I read this other question (I realize I'm not the first one asking this) where it says that SMTP is a “store and forward” protocol (meaning that it may not try to send the message immediately) which I also understand... but at the same time I find contradictory.

How can asdf.asdf store and forward a message when it doesn't even exist? Why does http://verify-email.org/ (and other web services, I didn't try only this one) get an immediate response while SmtpClient does not? What are they doing differently?

Answer 1

Let's understand how the mail flows (overly simplified):

Your App > Your Mail Server > Destination Mail Server > Recipient
               |                                          ^
               |                                          |
               + ---- if recipient on the same server ----+

So, when you send a mail to a recipient who is on the same server, for example if in the same domain as your Exchange Server is internally, then the Server knows if your recipient exists or not, and can immediately respond back to you.

However, when you send a mail to an external recipient, then your server will forward the email to the mail server (identified by the MX record of the domain name). When this happens, two things happen:

1. You server gladly accepts it from you, and sends you an accepted for delivery ack immediately
2. Then it tries to forward the mail to the external server based on its current queue

It does not keep your app waiting indefinitely until the queue clears and it actually sends the message. If it were to do that, there are chances your app could be waiting for as long as 4 days!

Regarding the other question/answer you linked to, they are assuming that the recipient is on the same server.

Normally, when you send mails using smtpClient through your app, you don't connect to the recipient's server directly. You use either a smart host, or your own mail server. You authenticate against your server which will then relay your mail. The methods of using VRFY and/or RCPT TO will work against your server. As long as the recipient is on the same server, you (smtpClient for that matter) will work against your own server and which is in a position to respond. However, when you are relaying your mail for an external server, your server will only accept your mail and cannot respond.

If, you need a response then you will have to connect to the destination server directly and issue SMTP commands there. In this case, you may or may not receive a response. This has also been clearly answered in the thread(s) you linked to. As an anti-spam measure, nearly all of the servers employ a tarpitting (in other words a delay or black hole) to protect from enumeration of its users.

It doesn't matter if you are sending mail to "someone@example.com" or "noone@nowhere.xyz", as long as it is an external server. The illustration I gave above is overly simplified. In reality, there could be several intermediary services which resolve the DNS for your email server.

You gave the example of "http://verify-email.org/". But, did you read what it says there on its home page?

This email verification tool actually connects to the mail server and checks whether the mailbox exists or not.

Which, means it is directly connecting to the MX server of the domain you are trying to send a mail to. Even you can do that. Only difference is that in real life, our applications can't do that. You connect to your local smarthost / relay.

Further, the home page of that service says:

What is being verified: Format: "name@domain.xxx"; Valid domain: "somebody@new.york" is not valid; Valid user: verify if the user; and mailbox really exist

Which means, that before attempting to connect to the destination server, this service does a check on format (perhaps a regex) and then does a DNS lookup to check if the domain is valid or not and then tries to connect to the destination server only after that.

Remember, even this service will not be able to tell you exactly what you want about existence of a user for all email services. For instance, I have employed tarpitting on my email server and hence even this service will not be able to tell you whether I exist or not. My mail server will never respond. "verify-email.org" will always return ok to you for everything.

BTW: "verify-email.org" will more often than not return this to you:

550 5.7.1 Service unavailable; Client host [verify-email.org] blocked using Spamhaus;

This service is already blocked by several anti-spam service providers!!

Hope that helps.

.