OpenCSP failed with error code 2148073494

Question

We started seeing this exception occur intermittently when our Azure App Service was moved to an App Service Environment. We're using Identity Server 4, and the exception occurs during token signing. This can be seen in the call stack below.

The signing cert is never saved to a cert store, it's loaded from a database as a byte array:

new X509Certificate2(rawData, password,
                    X509KeyStorageFlags.MachineKeySet |
                    X509KeyStorageFlags.PersistKeySet);

These references appear to be the same issue:

.NET Core X509Certificate2.PrivateKey throws nte_bad_keyset error

https://github.com/dotnet/corefx/issues/2583

Edit: Initially we thought this was something specific to the App Service Env in Azure, but now we're seeing the exception in standard App Services. As a test we created an Azure web job that loads a certificate from a byte array and creates JWT tokens (which require a private key for signing). This repros the error. Also of note is the Azure Web job won't even run when certain flags are passed to the X509Certificate2 ctor (e.g. X509KeyStorageFlags.UserKeySet).

Stack trace is the same each time:

System.Security.Cryptography.CryptographicException:
   at Internal.NativeCrypto.CapiHelper.CreateProvHandle(CspParameters parameters, Boolean randomKeyContainer)
   at System.Security.Cryptography.RSACryptoServiceProvider.get_SafeProvHandle()
   at System.Security.Cryptography.RSACryptoServiceProvider.get_SafeKeyHandle()
   at System.Security.Cryptography.RSACryptoServiceProvider..ctor(Int32 keySize, CspParameters parameters, Boolean useDefaultKeySize)
   at System.Security.Cryptography.RSACryptoServiceProvider..ctor(CspParameters parameters)
   at Internal.Cryptography.Pal.CertificatePal.<>c.<GetRSAPrivateKey>b__59_0(CspParameters csp)
   at Internal.Cryptography.Pal.CertificatePal.GetPrivateKey[T](Func`2 createCsp, Func`2 createCng)
   at Internal.Cryptography.Pal.CertificatePal.GetRSAPrivateKey()
   at Internal.Cryptography.Pal.CertificateExtensionsCommon.GetPrivateKey[T](X509Certificate2 certificate, Predicate`1 matchesConstraints)
   at Microsoft.IdentityModel.Tokens.X509SecurityKey.get_PrivateKey()
   at Microsoft.IdentityModel.Tokens.X509SecurityKey.get_HasPrivateKey()
   at Microsoft.IdentityModel.Tokens.AsymmetricSignatureProvider.HasPrivateKey(SecurityKey key)
   at Microsoft.IdentityModel.Tokens.AsymmetricSignatureProvider..ctor(SecurityKey key, String algorithm, Boolean willCreateSignatures)
   at Microsoft.IdentityModel.Tokens.CryptoProviderFactory.CreateSignatureProvider(SecurityKey key, String algorithm, Boolean willCreateSignatures)
   at Microsoft.IdentityModel.Tokens.CryptoProviderFactory.CreateForSigning(SecurityKey key, String algorithm)
   at System.IdentityModel.Tokens.Jwt.JwtSecurityTokenHandler.CreateEncodedSignature(String input, SigningCredentials signingCredentials)
   at System.IdentityModel.Tokens.Jwt.JwtSecurityTokenHandler.WriteToken(SecurityToken token)
   at IdentityServer4.Services.DefaultTokenCreationService.CreateJwtAsync(JwtSecurityToken jwt)
   at IdentityServer4.Services.DefaultTokenCreationService.<CreateTokenAsync>d__3.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at IdentityServer4.Services.DefaultTokenService.<CreateSecurityTokenAsync>d__9.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at IdentityServer4.ResponseHandling.TokenResponseGenerator.<CreateAccessTokenAsync>d__10.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at IdentityServer4.ResponseHandling.TokenResponseGenerator.<ProcessTokenRequestAsync>d__8.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at IdentityServer4.ResponseHandling.TokenResponseGenerator.<ProcessAsync>d__6.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at IdentityServer4.Endpoints.TokenEndpoint.<ProcessTokenRequestAsync>d__6.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at IdentityServer4.Endpoints.TokenEndpoint.<ProcessAsync>d__5.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at IdentityServer4.Hosting.IdentityServerMiddleware.<Invoke>d__3.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at IdentityServer4.Hosting.FederatedSignOutMiddleware.<Invoke>d__6.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at IdentityServer4.Hosting.AuthenticationMiddleware.<Invoke>d__2.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware`1.<Invoke>d__18.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware`1.<Invoke>d__18.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at Microsoft.AspNetCore.Cors.Infrastructure.CorsMiddleware.<Invoke>d__7.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at IdentityServer4.Hosting.BaseUrlMiddleware.<Invoke>d__2.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware`1.<Invoke>d__18.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware`1.<Invoke>d__18.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware`1.<Invoke>d__18.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware`1.<Invoke>d__18.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.ApplicationInsights.AspNetCore.ExceptionTrackingMiddleware.<Invoke>d__4.MoveNext()

Answer 1

Maybe you didn't give IIS access to your private keys. I had same error, but it was necessary to use X509 Certs instead of keys. I've just granted a permission to read private keys from cert for IIS_IUSRS on MMC. It helps for me.

There's a full answer from @thames: How to give ASP.NET access to a private key in a certificate in the certificate store?

UPD:

Step-by-step guide:

1. Create/purchase cert with private key
2. Import cert to "Local Computer" account (not "Current User"). Check "Allow private key to be exported" in Import Wizard.
3. Open MMC Console and add Certificates Snap-In. Choose "Computer Account".
4. Find your cert and right-click on it. Choose "All Tasks" > "Manage Private Keys"
5. Add Read access to your key for user IIS_IUSRS or IIS AppPool\<AppPoolName> (depends on IIS version and config. It's better to use "Advanced" button to show all possible options)
6. Restart your application

Answer 2

To resolve the issue we switched from X509 certificates (X509Certificate2) to RSA generated keys (RSACryptoServiceProvider). Identity Server 4 supports both. The exception no longer occurs.

Answer 3

That error means that somewhere after the certificate learned where the key was being stored it got deleted.

• Maybe something is calling CngKey.Delete
• Maybe something is cloning it into an RSACryptoServiceProvider and setting PersistKeyInCsp to false.
• Maybe something is just cleaning up the machine keys directory after the fact.
• The other (very rare) known case is where two threads load the PFX in parallel (they don't have to be in the same process) with one of them not having PersistKeySet asserted at import time.

If you're loading and discarding the certificates you really shouldn't set PersistKeySet. Every time you open a PFX another file is created on disk for each private key contained therein... those files get cleaned up normally (as long as the process doesn't abnormally terminate), but PersistKeySet prevents that cleanup from happening.

If nothing in code analysis reveals the two obvious key deletion locations (PersistKeyInCsp=false being the sneaky one) then you'll have to follow the filesystem auditing recommendations in .NET Core X509Certificate2.PrivateKey throws nte_bad_keyset error.