SslStream AuthenticateAsClient method getting "The handshake failed due to an unexpected packet format" error

Question

Our code uses TcpClient and SslStream in C# encounter an error The handshake failed due to an unexpected packet format. when calling AuthenticateAsClient method, the SMTP server is smtp.office365.com, we tried many ways but all not working.

1. Tried: System.Net.ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;

2. Tried: ssl.AuthenticateAsClient(Server, null, System.Security.Authentication.SslProtocols.Tls12, true);

3. Tried other AuthenticateAsClient overloading methods all not working.

4. Also with the tool Microsoft Network Monitor 3.4, we can confirm the Client Hello handshake use the TSL1.2 protocol.

Anyone had encountered a similar issue?

Below is the full code in the console with .NET Framework 8

    class Program
    {
        static void Main(string[] args)
        {
            try
            {
                string Server = "smtp-legacy.office365.com";
                //This server raise error too
                Server = "smtp.office365.com";
                int port = 587;
                //tried this, not working
                //System.Net.ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
                //
                using (var tcp = new System.Net.Sockets.TcpClient())
                {
                    tcp.Connect(Server, port);

                    System.Net.Security.RemoteCertificateValidationCallback remoteCertCallback =
                        new System.Net.Security.RemoteCertificateValidationCallback((Object sender,
                               System.Security.Cryptography.X509Certificates.X509Certificate cert,
                               System.Security.Cryptography.X509Certificates.X509Chain chain,
                               System.Net.Security.SslPolicyErrors Errors) => true);

                    //System.Net.Security.SslStream ssl = 
                    //    new System.Net.Security.SslStream(tcp.GetStream(), false, remoteCertCallback, null);

                    //Try set LocalCertificateSelectionCallback
                    System.Net.Security.SslStream ssl =
                        new System.Net.Security.SslStream(tcp.GetStream(), false, remoteCertCallback, SelectLocalCertificate);

                    //Raise error: The handshake failed due to an unexpected packet format.
                    #region Try to specify the protocol, not working
                    ssl.AuthenticateAsClient(Server, null, System.Security.Authentication.SslProtocols.Tls12, true);
                    #endregion

                    #region Try to set local certificate, not working
                    //tried to set local certificate, not working
                    //X509CertificateCollection clientCertificates = GetLocalCertificates();
                    //ssl.AuthenticateAsClient(Server, clientCertificates, System.Security.Authentication.SslProtocols.Tls12, true);
                    #endregion

                    #region Default way, not working
                    //Raise error: The handshake failed due to an unexpected packet format.
                    //ssl.AuthenticateAsClient(Server);
                    #endregion


                }

            }
            //Raise error: The handshake failed due to an unexpected packet format.
            catch (Exception ex)
            {
                Console.WriteLine(ex.ToString());
            }

            Console.Read();

        }


        public static X509Certificate2Collection GetLocalCertificates()
        {
            X509Certificate2Collection Certificates = null;
            // Read the certificate from the store
            X509Store store = new X509Store(StoreName.My, StoreLocation.LocalMachine);
            try
            {
                store.Open(OpenFlags.ReadOnly);
                //Certificates = store.Certificates.Find(X509FindType.FindBySubjectDistinguishedName,
                //"CN=[YOUR DOMAIN]", false);
                Certificates = store.Certificates;
            }
            finally
            {
                store.Close();
            }

            return Certificates;
        }

        public static X509Certificate SelectLocalCertificate(
            object sender,
            string targetHost,
            X509CertificateCollection localCertificates,
            X509Certificate remoteCertificate,
            string[] acceptableIssuers)
        {
            if (acceptableIssuers != null &&
                acceptableIssuers.Length > 0 &&
                localCertificates != null &&
                localCertificates.Count > 0)
            {
                // Use the first certificate that is from an acceptable issuer.
                foreach (X509Certificate certificate in localCertificates)
                {
                    string issuer = certificate.Issuer;
                    if (Array.IndexOf(acceptableIssuers, issuer) != -1)
                        return certificate;
                }
            }
            if (localCertificates != null &&
                localCertificates.Count > 0)
                return localCertificates[0];

            return null;
        }

    }

Answer 1

Finally, we found the solution, post here just in case anyone has the same issue:

So there are two cases:
Case 1: For TLS, need to call AuthenticateAsClient after step 5.
Case 2: For SSL, need to call AuthenticateAsClient after step 1.

Our issue was we called AuthenticateAsClient after step 1 for TLS, the correct position is we should call AuthenticateAsClient after Step 5 that after sending StartTLS that make sure the server return information that supports StartTSL.

Step 1. Client: TCP handshake/connection
(For SSL only) call AuthenticateAsClient before EHLO
Step 2. Server: 200 Ready
Step 3. Client: EHLO
Step 4. Server: 250 StartTLS(means support TLS)
Step 5. Client: StartTLS(send StartTLS commend)
(For TLS only) call AuthenticateAsClient after StartTLS
Step 6. Server: 220 Go Head
Step 7. Client: sending data/email with encryption (edited)