CODEWITHSUNDEEP
azureasp.net-coreazure-web-app-serviceazure-keyvault
zola25
zola25

Reputation: 1901

KeyVaultErrorException: Operation returned an invalid status code 'Forbidden'

I'm trying to set up my web app, hosted in Azure to read settings from Azure KeyVault.

I've been following this guide: https://anthonychu.ca/post/secrets-aspnet-core-key-vault-msi/

The example shows how to access app settings from KeyVault with the configuration:

public static IWebHost BuildWebHost(string[] args) =>
    WebHost.CreateDefaultBuilder(args)
     .ConfigureAppConfiguration((ctx, builder) =>
     {
         var keyVaultEndpoint = Environment.GetEnvironmentVariable("KEYVAULT_ENDPOINT");
         if (!string.IsNullOrEmpty(keyVaultEndpoint))
         {
             var azureServiceTokenProvider = new AzureServiceTokenProvider();
             var keyVaultClient = new KeyVaultClient(
                 new KeyVaultClient.AuthenticationCallback(
                     azureServiceTokenProvider.KeyVaultTokenCallback));
             builder.AddAzureKeyVault(
                 keyVaultEndpoint, keyVaultClient, new DefaultKeyVaultSecretManager());
         }
     })
    .UseApplicationInsights()
    .UseStartup<Startup>()
    .Build();

I've added the KEYVAULT_ENDPOINT environment variable to the application settings. I've enabled MSI on the app service, and I've authorized my Azure User, and my application, from the Key Vault Access Policies:

enter image description here

With Get and List Operations:

enter image description here

And I've added the secret to key vault. Running locally, I can access the secret.

But my ASP .NET Core site fails on startup with this in the stdout logs:

Unhandled Exception: Microsoft.Azure.KeyVault.Models.KeyVaultErrorException: Operation returned an invalid status code 'Forbidden'
   at Microsoft.Azure.KeyVault.KeyVaultClient.GetSecretsWithHttpMessagesAsync(String vaultBaseUrl, Nullable`1 maxresults, Dictionary`2 customHeaders, CancellationToken cancellationToken)
   at Microsoft.Azure.KeyVault.KeyVaultClientExtensions.GetSecretsAsync(IKeyVaultClient operations, String vaultBaseUrl, Nullable`1 maxresults, CancellationToken cancellationToken)
   at Microsoft.Extensions.Configuration.AzureKeyVault.AzureKeyVaultConfigurationProvider.LoadAsync()
   at Microsoft.Extensions.Configuration.AzureKeyVault.AzureKeyVaultConfigurationProvider.Load()
   at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers)
   at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
   at Microsoft.AspNetCore.Hosting.WebHostBuilder.BuildCommonServices(AggregateException& hostingStartupErrors)
   at Microsoft.AspNetCore.Hosting.WebHostBuilder.Build()
   at Blog.Program.BuildWebHost(String[] args) in D:\a\1\s\[csproj name]\Program.cs:line 22
   at [csproj name].Program.Main(String[] args) in D:\a\1\s\[csproj name]\Program.cs:line 16

I've checked that MSI_ENDPOINT and MSI_SECRET environment variables exist by calling SET from the debug console. enter image description here

I can also see the KEYVAULT_ENDPOINT variable.

Any suggestions what could be going wrong or on what to try next? Since it works locally it must be an authentication issue, but I believe it's authenticating locally with my azure User that I've authorized in key vault, rather than as the Azure App Service.

Upvotes: 34

Views: 94280

Answers (9)

Jim Cargle
Jim Cargle

Reputation: 101

If you added the access policy while you were logged into Azure through Visual Studio, you'll have to log out and log back in again to refresh your permissions.

Upvotes: 0

Carlos Cruz
Carlos Cruz

Reputation: 505

In the AZ KV, access policies... by selecting only the principal should work.

enter image description here

Upvotes: 1

sargeMonkey
sargeMonkey

Reputation: 696

I want to add another case here with the same symptoms/error messages. Azure keyvault returned Forbidden when one of the secrets was disabled.

I had all the required permissions - I was accessing keyvault with Get and List granted for secrets (I was using only secrets) for the Managed Identity of my App Service.

It was confusing to get the "Access Forbidden" error, and it took some experimentation to figure out the issue. Hope this helps someone who have faced the same issue.

Upvotes: 3

Victory King
Victory King

Reputation: 1

Another possibility is that the service principal needs API permissions for Key Vault. In Azure portal, go to Service Principal, then choose API permissions on the left hand menu. Then click Add permissions. Here you can choose Key Vault and save. This will allow the code to retrieve the key vault secrets.

Upvotes: 0

Mark Szabo
Mark Szabo

Reputation: 10488

For me to resolve the issue, I had to add the IP addresses of the web app to the firewall as not all of them were getting added when you select "Allow trusted Microsoft services..."

outbound ips of the web app

Azure Key Vault firewall settings

Thanks to: https://azidentity.azurewebsites.net/post/2019/01/03/key-vault-firewall-access-by-azure-app-services

Upvotes: 4

Ralph Willgoss
Ralph Willgoss

Reputation: 12163

Check to ensure that the Service Principal you are using has been added and has the appropriate permissions for the operations you want to perform (Keys/Secrets etc):

enter image description here

Check to see if the KeyVault has restricted network access, toggle to test if this is the issue or not:

enter image description here

Upvotes: 3

Viral Jain
Viral Jain

Reputation: 56

only select principal and add web app,don't select authorized application enter image description here

Upvotes: 3

JleruOHeP
JleruOHeP

Reputation: 10376

It is dodgy, but the problem with your setup is that you enabled "Authorized application" when you created the Access policy for your KeyValut.

You can see it because you have "Application+Application" on your screenshot. I imagine, you created the policy with your webapp being both principal and authrorized application. This kinda creates a loop.

To make this setup work just delete your existing policy and create a new one, where you select only the Principal:

enter image description here

Upvotes: 40

Svek
Svek

Reputation: 12848

I have seen issues like this before, with development and production systems not behaving the same way when dealing with environment variables.

My first area to look at in these scenarios is to check if you have properly prefixed your environment variables with ASPNETCORE_ in your production system. This prefix is the default prefix for ASP.NET Core web hosts.

| Key                                |  Value                     |
|====================================|============================|
| ASPNETCORE_KEYVAULT_ENDPOINT       |  xxxxxxxxxxxxxxxx          |
| ASPNETCORE_MSI_ENDPOINT**          |  xxxxxxxxxxxxxxxx          |
| ASPNETCORE_MSI_SECRET**            |  xxxxxxxxxxxxxxxx          |

** Note that in your code I don't ever see you using MSI_ENDPOINT or MSI_SECRET. I only see you using KEYVAULT_ENDPOINT

If you want to explicitly set the prefix to something else, the prefix you want to use can be passed as an argument when initializing the web host via the configuration.

.AddEnvironmentVariables("ASPNETCORE_"); // choose your own prefix here

Upvotes: 1

Related Questions