Xamarin Forms android application cannot reach the backend on devices running Android OS 13 or earlier after migrating backend to Cloudflare

Question

I know, I know, dinosaur tech. But I have been tasked to figure this out. Any help would receive my eternal gratitude...

Over the weekend, my company migrated our server security to Cloudflare. Forgive my ignorance, I am not well versed in network stuff and I was not involved in this process. That said, when I came in on Monday, all hell broke loose when reports of devices running our app receiving "Connection Errors" started coming in en masse.

Upon troubleshooting, I was able to determine the following:

• Issue only occurs on Android Devices, and only on Android devices running Android OS 13 or earlier.
• The error occurs at the point in the code where the request is sent, and throws a System.Net.Http.HttpRequestException with message "The SSL connection could not be established, see inner exception." (stack trace below)
• Upon speaking to network and server administrator, no requests from said IP to said backend are being rejected by cloudflare, nor are there any failed request logs in our IIS logs.
• Updating the Device Android OS to Android 14 resolves the issue.

Is there something about the TLS/SSL Registry on an app utilizing Xamarin Forms with an older Android OS that cloud flare is rejecting? Any idea on how this can be resolved without forcing the user to update their OS which clients are pushing back on hard?

STACK TRACE:

System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. ---> Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED at /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/boringssl/ssl/handshake_client.c:1132 at Mono.Btls.MonoBtlsContext.ProcessHandshake () [0x00042] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/mcs/class/System/Mono.Btls/MonoBtlsContext.cs:220 at Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake (Mono.Net.Security.AsyncOperationStatus status, System.Boolean renegotiate) [0x000da] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/mcs/class/System/Mono.Net.Security/MobileAuthenticatedStream.cs:715 at (wrapper remoting-invoke-with-check) Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake(Mono.Net.Security.AsyncOperationStatus,bool) at Mono.Net.Security.AsyncHandshakeRequest.Run (Mono.Net.Security.AsyncOperationStatus status) [0x00000] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/mcs/class/System/Mono.Net.Security/AsyncProtocolRequest.cs:289 at Mono.Net.Security.AsyncProtocolRequest.ProcessOperation (System.Threading.CancellationToken cancellationToken) [0x000fc] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/mcs/class/System/Mono.Net.Security/AsyncProtocolRequest.cs:223 --- End of inner exception stack trace --- at Mono.Net.Security.MobileAuthenticatedStream.ProcessAuthentication (System.Boolean runSynchronously, Mono.Net.Security.MonoSslAuthenticationOptions options, System.Threading.CancellationToken cancellationToken) [0x0025c] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/mcs/class/System/Mono.Net.Security/MobileAuthenticatedStream.cs:310 at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore (System.IO.Stream stream, System.Net.Security.SslClientAuthenticationOptions sslOptions, System.Threading.CancellationToken cancellationToken) [0x0007b] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/ConnectHelper.cs:165 --- End of inner exception stack trace ---

Uninstalling and reinstalling, restarting the device, and switching networks does not resolve the issue. The application is a dinosaur and was utilizing Xamarin.Forms and targets Android 11, but had no issues prior to the Cloudflare migration.

I tried forcing the app to use TLS 1.2 by adding trying to add both of the following lines in the MainActivity.cs OnCreate method:

`System.Net.ServicePointManager.SecurityProtocol = System.Net.SecurityProtocolType.Tls12;`

And

`Java.Security.Security.SetProperty("jdk.tls.client.protocols", "TLSv1.2");`

but it made no difference.

The only thing that resolves this is updating to Android 14 but I don't see anything in the Android 14 release notes that indicates that it might affect connections...

Unfortunately I can't ask all clients to upgrade as many have devices that either cannot upgrade or will be quite upset at this ask :(

UPDATE August 9th 2024: it appears to be a certificate chain issue. I am able to create a custom http handler and see that it is an SSL Policy Error of type RemoteCertificateChainErrors. It is a certificate issued by "Let's Encrypt" and it appears that Android sometimes has problems with certificates issues by them...