Timeout on high amount of messages with system.net.mail on internal exchange smtp server

Question

I'm working on a program to send weekly reminders / updates to each sales employee at our company. About 120 per week, automated, in a short amount of time. This is an internal program, to internal recipients only and I am not concerned about spam or unwanted messages and unsubscription is not an option. Each message is personalized for each sales staff member based on their customer list and contains weekly "to do" items for them to call customers.

I'm running Exhange 2007 internally with an open internal SMTP connector for both my development environment and the server that runs these automation projects. The program is working fine for one or two stores but when I run it for every store I get a time out somewhere after 110 messages have been sent.

I'm not attempting to queue mail or hold them in blocks, as I iterate through the sale person list, and do each respective lookup and message build, I am attempting to send messages with the following sub.

Sub doMail(ByRef MessageBody As String, ByVal nameString As String, ByVal sendTo As ArrayList, Optional ByVal markurgent As Boolean = False, _
                 Optional ByVal sendCC As ArrayList = Nothing, Optional ByVal sendBcc As ArrayList = Nothing)
    Try
        ' Setup Mail Message
        Dim oClient As SmtpClient = New SmtpClient(ConfigurationManager.AppSettings("mail_server").ToString())
        oClient.Timeout = 20000
        'oClient.Port = 50747
        Dim objMessage As New MailMessage()
        objMessage.From = New System.Net.Mail.MailAddress(ConfigurationManager.AppSettings("mail_from_address").ToString, ConfigurationManager.AppSettings("mail_from_name").ToString)
        objMessage.Subject = String.Format("Weekly Hitlist Report ~ {0}", nameString)

        If (ConfigurationManager.AppSettings("debugMode").ToString() = 1) Then
            Dim debugSB As StringBuilder = New StringBuilder

            For Each s As String In sendTo
                debugSB.AppendLine(String.Format("To: {0}<br>", s))
            Next
            For Each s As String In sendCC
                debugSB.AppendLine(String.Format("cc: {0}<br>", s))
            Next
            For Each s As String In sendBcc
                debugSB.AppendLine(String.Format("bcc: {0}<br>", s))
            Next
            MessageBody = String.Format("Debug Mode is Active. <br> {0} <br><hr noshade>{1}", debugSB.ToString, MessageBody)
            objMessage.To.Add(ConfigurationManager.AppSettings("mail_to_address").ToString)
        Else
            For Each receiver As String In sendTo
                Try
                    objMessage.To.Add(receiver.Trim())
                Catch ex As Exception
                    ' do nothing
                End Try
            Next
            For Each receiver As String In sendCC.ToArray
                Try
                    objMessage.CC.Add(receiver.Trim())
                Catch ex As Exception
                    ' do nothing
                End Try
            Next
            For Each receiver As String In sendBcc
                Try
                    objMessage.Bcc.Add(receiver.Trim())
                Catch ex As Exception
                    ' do nothing
                End Try
            Next
        End If



        objMessage.Body = MessageBody
        objMessage.Priority = IIf(markurgent, MailPriority.High, MailPriority.Normal)
        objMessage.IsBodyHtml = True
        oClient.Send(objMessage)
        oClient = Nothing
        Console.WriteLine(String.Format("{1} - message sent - {0} ", nameString, Now()))
    Catch ex As Exception
        Console.WriteLine(ex.Message)
        Console.WriteLine("****************")
        Console.WriteLine(ex.StackTrace)
    End Try
End Sub

I have attempted to increase the client time out to 20 seconds and I'm explicitly attempting to close out the client connection to the mail server after each message by setting it to nothing.

Everything LOOKS to proceed OK until the very end, I have removed our sales person names from the below output snippet. Right now everything is coming to me since I'm running this in debug mode, we have not attempted a live run to the entire company yet so I don't know if it is since everything is going to one recipient.

After I get the error (connection time out) and the system resumes, I get any messages that are sent after the exception and only AFTER the program terminates. In prior versions of .NET I know that programs had to exit or the thread had to end before messages would be sent but other programs I have going I don't generally have this in .NET 4 any more.

Output including error; first part has been snipped

 ...
 4/15/2013 9:46:36 AM - message sent - 
 4/15/2013 9:46:41 AM - message sent - 
 4/15/2013 9:46:46 AM - message sent - 
 4/15/2013 9:46:51 AM - message sent - Store 14 Unassigned
 4/15/2013 9:46:56 AM - message sent - 
 4/15/2013 9:47:01 AM - message sent - 
 4/15/2013 9:47:06 AM - message sent - 
 4/15/2013 9:47:11 AM - message sent - 
 4/15/2013 9:47:16 AM - message sent - 
 4/15/2013 9:47:21 AM - message sent - 
 4/15/2013 9:47:26 AM - message sent - 
 4/15/2013 9:47:31 AM - message sent - 
 4/15/2013 9:47:36 AM - message sent - 
 4/15/2013 9:47:41 AM - message sent - 
 Service not available, closing transmission channel. The server response was: 4.4.1 Connection timed out
 ****************
    at System.Net.Mail.MailCommand.CheckResponse(SmtpStatusCode statusCode, String response)
    at System.Net.Mail.MailCommand.Send(SmtpConnection conn, Byte[] command, String from)
    at System.Net.Mail.SmtpTransport.SendMail(MailAddress sender, MailAddressCollection recipients, String deliveryNotify, SmtpFailedRecipientException& exception)
    at System.Net.Mail.SmtpClient.Send(MailMessage message)
    at HitListRunner_Main.Module1.doMail(String& MessageBody, String nameString, ArrayList sendTo, Boolean markurgent, ArrayList sendCC, ArrayList sendBcc) in C:\Users\markl\Documents\Visual Studio 2010\Projects\HitListRunner\HitListRunner-Main\Module1.vb:line 391
 4/15/2013 9:47:46 AM - message sent - Store 20 Unassigned
 4/15/2013 9:47:51 AM - message sent - Store 98 Unassigned
 4/15/2013 9:47:58 AM - message sent - !! - UNDETERMINED LIST - !!
 ending - enter to exit

I have also tried sending for each individual store and smaller subsets. I only receive this error when I run for the entire company all at one time.

I don't see anything in Exchange prohibiting this and I don't see a high message queue waiting. Any suggestions on a different method to send the messages or something else to try? (I would prefer not to use a 3rd party dll or outside component).

Answer 1

I've experienced the pain of this issue. After 2 years of research I've not been able to pinpoint an exact cause of the issue with my .net application. There are a number of issues and implementation options that will help minimize the impact of this exception. We are now at a point this problem doesn't impact the end users in a noticeable way.

1. You will need to ensure some of the SMTP configuration variables are set in the *.config to help tune the application to your environment. SMTP Client Timeout, Retry Attempts, Time between retries etc. We found about 20 seconds between attempts and about 5 attempts before completely stopping worked.
2. You will need to implement a logging mechanism to record when and how frequently they occur, so you can set the variables above. I was looking for volume of emails, time of day, service pack installations, virus checker or firewall interference from the machines and servers event log, to try and find a cause.
3. Implement a non Blockingcollection to manage the sending of all the emails, because if your sending a large volume via code, your going to run into the SMTP 4.4.1 timeout error. This a producer consumer pattern and I've found this a relatively easy implementation to implement and very successful for this issue. The queue allows you to ensure the emails will get sent eventually. If you want a full fail safe you can persist that queue to disk as well. We never need to fall back onto that piece of code.
4. Faking or Mocking the .net SMTP implementation to unit test your code for this issue was not easy to do without third party tools. I didn't go down this path, because at the end of the day there were too many parts outside of my control. Basically I'm just going to assume it will always fail and I place lots of fallback options to wait for the problem to clear itself.

Things we tried that that didn't seem to make a difference to the frequency of 4.4.1 timeout exceptions.

1. Throttling the rate the emails get sent to exchange.
2. Changing the time of day they get sent (aka out of hours)
3. Batching the emails into small blocks and then clearing all the .net components from memory via garbage collection.

Hope that helps.

Code Snippits

''' <summary>
''' Provide a mechanism to throttle the subsequent attempts to send emails.
''' </summary>
''' <param name="emailAddress"></param>
''' <param name="attemptNumber"></param>
''' <remarks>Need to implement a more dynamic function as attempts persist, pause for longer for a number of attempt, then add the email to a failed queue.</remarks>
Private Shared Sub RestBeforeSendingAgain(ByVal emailAddress As String, ByVal attemptNumber As Integer)

    If attemptNumber > 1 Then

        '   Wait 30 seconds in between every attempt, or the value configured in the config file.
        Dim retryIntervalInSeconds As Integer = 30
        If Not String.IsNullOrEmpty(System.Configuration.ConfigurationManager.AppSettings.Item("SMTPFailureReTryIntervalInSeconds")) Then
            Integer.TryParse(System.Configuration.ConfigurationManager.AppSettings.Item("SMTPFailureReTryIntervalInSeconds"), retryIntervalInSeconds)
        End If
        logger.Warn("Problem emailing {0}. Waiting {1} seconds to help clear technical issues.", emailAddress, retryIntervalInSeconds.ToString)
        System.Threading.Thread.Sleep(System.TimeSpan.FromSeconds(retryIntervalInSeconds))

    End If
End Sub

    ''' <summary>
''' Attempt to send an email message and record any issues with that mailmessage to the application log.
''' Allows the process to know the success or failure of that approach.
''' </summary>
''' <param name="mailClient"></param>
''' <param name="mailMessage"></param>
''' <param name="attemptNumber"></param>
''' <returns>The success or failure of the SMTP server to send the message without causing an error.</returns>
''' <remarks></remarks>
Private Shared Function AttemptToSendEmail(ByRef mailClient As System.Net.Mail.SmtpClient, ByRef mailMessage As MailMessage, ByVal attemptNumber As Integer) As Boolean
    Dim result As Boolean = False
    Try
        logger.Trace("Started: AttemptToSendEmail", attemptNumber)

        RestBeforeSendingAgain(mailMessage.To.ToString, attemptNumber)
        mailClient.Send(mailMessage)
        result = True

    Catch smtpEx As SmtpException
        result = False
        logger.ErrorException(String.Format("{3}{1}{0}{1}{2}", ExceptionFormater.FormatException(smtpEx), Environment.NewLine, ExceptionFormater.FormatStack(smtpEx), "AttemptToSendEmail failed with Smtp exception."), smtpEx)

    Catch ex As Exception
        result = False
        logger.ErrorException(String.Format("{3}{1}{0}{1}{2}", ExceptionFormater.FormatException(ex), Environment.NewLine, ExceptionFormater.FormatStack(ex), "AttemptToSendEmail failed with generic exception."), ex)

    Finally

        If result Then
            logger.Trace("Succeeded: To email {0} on attempt {1}.", mailMessage.To.ToString, attemptNumber)
        Else
            logger.Warn("Failed: To email {0} on attempt {1}.", mailMessage.To.ToString, attemptNumber)
        End If

    End Try

    Return result

End Function

'   Configure the timeout in milliseconds from the App.Config file.
'   200 seconds = 200,000 miliseconds
MySmtpClient.Timeout = CInt(TimeSpan.FromSeconds(SmtpClientTimeoutSeconds).TotalMilliseconds)