random SSLHandshakeException when calling https://outlook.office365.com/EWS/Exchange.asmx from Java application

Question

I have a small application Java 11 that runs in a container and connects to Exchange online on https://outlook.office365.com/EWS/Exchange.asmx . I am connecting to it using OKHttp3 (v4.9.1).

Most of the times, it works.. but from time to time, it will fail with below exception :

Caused by: javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake at java.base/sun.security.ssl.SSLSocketImpl.handleEOF(SSLSocketImpl.java:1321) at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1160) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1063) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:402) at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.kt:379) at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.kt:337) at okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:209) at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:226) at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:106) at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:74) at okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:255) at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) at okhttp3.logging.HttpLoggingInterceptor.intercept(HttpLoggingInterceptor.kt:221) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201) at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154) at vincent.email.kpi.tracker.o365.MyHttpClient.send(MyHttpClient.java:47) at com.microsoft.aad.msal4j.HttpHelper.executeHttpRequestWithRetries(HttpHelper.java:86) at com.microsoft.aad.msal4j.HttpHelper.executeHttpRequest(HttpHelper.java:64) ... 7 common frames omitted Caused by: java.io.EOFException: SSL peer shut down incorrectly at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:167) at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:108) at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1152) ... 31 common frames omitted

The fact that it's failing randomly is puzzling me... It's running in the same Docker image, so configuration and certificates on my side are not changing from one run to the other. I would expect that it's the same on office365 server.

Is there anything that I can add somewhere to make it more "deterministic" ?

Below is my code :

import com.microsoft.aad.msal4j.HttpMethod;
import com.microsoft.aad.msal4j.HttpRequest;
import com.microsoft.aad.msal4j.HttpResponse;
import com.microsoft.aad.msal4j.IHttpResponse;
import java.io.IOException;
import java.net.InetSocketAddress;
import java.net.Proxy;
import java.nio.charset.StandardCharsets;
import java.security.KeyManagementException;
import java.security.NoSuchAlgorithmException;
import java.security.cert.CertificateException;
import java.util.List;
import javax.net.ssl.SSLContext;
import javax.net.ssl.SSLSocketFactory;
import javax.net.ssl.TrustManager;
import javax.net.ssl.X509TrustManager;
import okhttp3.Credentials;
import okhttp3.Headers;
import okhttp3.OkHttpClient;
import okhttp3.Request;
import okhttp3.RequestBody;
import okhttp3.Response;
import okhttp3.Route;
import okhttp3.logging.HttpLoggingInterceptor;
import okhttp3.logging.HttpLoggingInterceptor.Level;

public class MyHttpClient implements com.microsoft.aad.msal4j.IHttpClient {

  public static final String PROXY_HOST = "my.proxy.url";
  public static final int PROXY_PORT = 8080;

  private final OkHttpClient client;

  public MyHttpClient(String authUser, String authPassword){
    this.client = buildHttpClient(authUser,authPassword);
  }

  @Override
  public IHttpResponse send(HttpRequest httpRequest) throws IOException {

    // Map URL, headers, and body from MSAL's HttpRequest to OkHttpClient request object
    var request = buildOkRequestFromMsalRequest(httpRequest);

    // Execute Http request with OkHttpClient
    var okHttpResponse= client.newCall(request).execute();

    // Map status code, headers, and response body from OkHttpClient's Response object to MSAL's IHttpResponse
    return buildMsalResponseFromOkResponse(okHttpResponse);
  }

  private static IHttpResponse buildMsalResponseFromOkResponse(Response okHttpResponse) throws IOException {

    var msal4j = new HttpResponse();
    msal4j.statusCode(okHttpResponse.code());

    for (String headerKey : okHttpResponse.headers().names()) {
      List<String> val = okHttpResponse.headers(headerKey);

      msal4j.headers().put(headerKey, val);
    }
    msal4j.body(okHttpResponse.body().string());

    return msal4j;
  }

  private static Request buildOkRequestFromMsalRequest(HttpRequest httpRequest) {

    var builder=new Request.Builder()
        .url(httpRequest.url());

    if(httpRequest.httpMethod()== HttpMethod.POST) {
      builder.method("POST", RequestBody.create(httpRequest.body().getBytes(StandardCharsets.UTF_8)));
    }
   //defaults to GET, with no body


    if(httpRequest.headers()!=null){
      builder.headers(Headers.of(httpRequest.headers()));
    }

    return builder.build();
  }


  private static OkHttpClient buildHttpClient(final String authUser, final String authPassword) {

    okhttp3.Authenticator proxyAuthenticator = new okhttp3.Authenticator() {

      @Override
      public Request authenticate(Route route, Response response) throws IOException {
        String credential = Credentials.basic(authUser, authPassword);
        return response.request().newBuilder()
            .header("Proxy-Authorization", credential)
            .build();
      }
    };

    var logging = new HttpLoggingInterceptor();
    logging.setLevel(Level.BODY);

    return new OkHttpClient.Builder()
        .proxy(new Proxy(Proxy.Type.HTTP, new InetSocketAddress(PROXY_HOST, PROXY_PORT)))
        .proxyAuthenticator(proxyAuthenticator)
        //TODO should no ignore certificates in prod..
        // --> certificate should be added in store
        .sslSocketFactory(trustAllSslSocketFactory, (X509TrustManager)trustAllCerts[0])
        .addInterceptor(logging)
        .hostnameVerifier((hostname, session) -> true)
        .build();
  }

  private static final TrustManager[] trustAllCerts = new TrustManager[] {
      new X509TrustManager() {
        @Override
        public void checkClientTrusted(java.security.cert.X509Certificate[] chain, String authType)
            throws CertificateException {
        }

        @Override
        public void checkServerTrusted(java.security.cert.X509Certificate[] chain, String authType)
            throws CertificateException {
        }

        @Override
        public java.security.cert.X509Certificate[] getAcceptedIssuers() {
          return new java.security.cert.X509Certificate[]{};
        }
      }
  };

  private static final SSLContext trustAllSslContext;
  static {
    try {
      trustAllSslContext = SSLContext.getInstance("SSL");
      trustAllSslContext.init(null, trustAllCerts, new java.security.SecureRandom());
    } catch (NoSuchAlgorithmException | KeyManagementException e) {
      throw new Office365Exception(e);
    }
  }
  private static final SSLSocketFactory trustAllSslSocketFactory = trustAllSslContext.getSocketFactory();

}

Answer 1

I would expect that it's the same on office365 server.

I would expect there to be thousands of Office365 server instances with load balancers (etcetera) in the front of them. It is possible that each time your client attempts to connect that it is talking to a different SSL endpoint. They may not all be configured the same. That would be one possible explanation for the nondeterminism.

Is there anything that I can add somewhere to make it more "deterministic"?

Probably not.

But what you could (should) do is to try to figure out why remote server is resetting the connection.

Connection resets by the server during SSL connection handshakes typically mean that the SSL endpoint has decided that can't establish a secure connection. Common reasons include not being able to agree on the protocol version or crypto algorithms to use. (This typically happens when the client or server side is using SSL protocols or crypto algorithms that are no longer considered to be secure.)

The standard way to diagnose the problem is to run the JVM with the -Djavax.net.debug=all option. This will log a whole lot of information including the details of the SSL negotiation. You can compare what the client and server are "offering", then figure out what the mismatch is.

For more information; see Debugging SSL/TLS Connections.