RequestFailedException/403 Forbidden errors accessing Azure Key Vault with role based access policies

Question

I have a scenario where I'm getting the below exception when trying to debug an ASP.NET Core Web Application in Visual Studio that has connected services for Azure Key Vault and Azure Application Configuration resources connected to an App Service. The user that I'm signed into Visual Studio with appears to have the correct permissions on the KV and the same as another developer who does NOT get the error I do. I can also successfully view and inspect the values for all secrets in the Secrets page for the AKV while signed into Azure using the same user.

The error:

C:\Professional\Projects\Inventive\inventivegroup\mgr360>dotnet watch run watch : Started Unhandled exception. Azure.RequestFailedException: Service request failed. Status: 403 (Forbidden)

Content: {"error":{"code":"Forbidden","message":"Caller is not authorized to perform action on resource.\r\nIf role assignments, deny assignments or role definitions were changed recently, please observe propagation time.\r\nCaller: appid=c41ed4ac-ID-SNIP;oid=3108ce41-ID-SNIP;iss=https://sts.windows.net/359ccce3-ID-SNIP/\r\nAction: 'Microsoft.KeyVault/vaults/secrets/readMetadata/action'\r\nResource: '/subscriptions/666a80a4-ID-SNIP/resourcegroups/RESOURCEGROUPNAME/providers/microsoft.keyvault/vaults/webappvaultname'\r\nAssignment: (not found)\r\nVault: NameOfWebAppvault;location=eastus\r\n","innererror":{"code":"ForbiddenByRbac"}}}

Headers: Cache-Control: no-cache Pragma: no-cache x-ms-keyvault-region: eastus x-ms-client-request-id: bd5ef0e5-ID-SNIP x-ms-request-id: 8053b6d8-ID-SNIP x-ms-keyvault-service-version: 1.2.236.0 x-ms-keyvault-network-info: conn_type=Ipv4;addr=50.IP.SNIP;act_addr_fam=InterNetwork; x-ms-keyvault-rbac-cache: ra_age=0;da_age=7453;rd_age=7453;brd_age=11547;ra_notif_age=99;dec_lev=3; X-Powered-By: ASP.NET Strict-Transport-Security: max-age=31536000;includeSubDomains X-Content-Type-Options: nosniff Date: Fri, 23 Apr 2021 18:21:57 GMT Content-Length: 701 Content-Type: application/json; charset=utf-8 Expires: -1

at Azure.Security.KeyVault.KeyVaultPipeline.SendRequestAsync(Request request, CancellationToken cancellationToken) at Azure.Security.KeyVault.KeyVaultPipeline.GetPageAsync[T](Uri firstPageUri, String nextLink, Func1 itemFactory, String operationName, CancellationToken cancellationToken) at Azure.Core.PageResponseEnumerator.FuncAsyncPageable1.AsPages(String continuationToken, Nullable1 pageSizeHint)+MoveNext() at Azure.Core.PageResponseEnumerator.FuncAsyncPageable1.AsPages(String continuationToken, Nullable1 pageSizeHint)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult() at Azure.AsyncPageable1.GetAsyncEnumerator(CancellationToken cancellationToken)+MoveNext() at Azure.AsyncPageable1.GetAsyncEnumerator(CancellationToken cancellationToken)+MoveNext() at Azure.AsyncPageable1.GetAsyncEnumerator(CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult() at Azure.Extensions.AspNetCore.Configuration.Secrets.AzureKeyVaultConfigurationProvider.LoadAsync() at Azure.Extensions.AspNetCore.Configuration.Secrets.AzureKeyVaultConfigurationProvider.LoadAsync() at Azure.Extensions.AspNetCore.Configuration.Secrets.AzureKeyVaultConfigurationProvider.Load() at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers) at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build() at FOO_Web_App.Program.<>c.b__1_2(WebHostBuilderContext hostingContext, IConfigurationBuilder config) in C:\Professional\Projects\FOO\Program.cs:line 37 at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.<>c__DisplayClass9_0.b__0(HostBuilderContext context, IConfigurationBuilder builder) at Microsoft.Extensions.Hosting.HostBuilder.BuildAppConfiguration() at Microsoft.Extensions.Hosting.HostBuilder.Build() at FOO_Web_App.Program.Main(String[] args) in C:\Professional\Projects\FOO\Program.cs:line 17

watch : Exited with error code -532462766

The code (which also occurs running 'dotnet watch run'):

public class Program
{
    public static void Main(string[] args)
    {
        CreateHostBuilder(args).Build().Run();
    }

    public static IHostBuilder CreateHostBuilder(string[] args) =>
        Host.CreateDefaultBuilder(args)
            .ConfigureAppConfiguration((context, config) =>
            {
                var keyVaultEndpoint = new Uri(Environment.GetEnvironmentVariable("VaultUri"));
                config.AddAzureKeyVault(
                    keyVaultEndpoint,
                    new DefaultAzureCredential());
            })
            .ConfigureWebHostDefaults(webBuilder =>
                webBuilder.ConfigureAppConfiguration((hostingContext, config) =>
                {
                    //Error here at config.Build():
                    //Azure.RequestFailedException: 'Service request failed.
                    //Status: 403(Forbidden)
                    //Content:
                    //{"error":{"code":"Forbidden","message":"Caller is not authorized to perform action on resource.\r\nIf role assignments, deny assignments or role definitions were changed recently, please observe propagation time.\r\nCaller: appid=c41ed4ac-ID-SNIP;oid=3108ce41-ID-SNIP;iss=https://sts.windows.net/359ccce3-ID-SNIP/\r\nAction: 'Microsoft.KeyVault/vaults/secrets/readMetadata/action'\r\nResource: '/subscriptions/666a80a4-ID-SNIP/resourcegroups/RESOURCEGROUPNAME/providers/microsoft.keyvault/vaults/webappvaultname'\r\nAssignment: (not found)\r\nVault: NameOfWebAppvault;location=eastus\r\n","innererror":{"code":"ForbiddenByRbac"}}}
                    var settings = config.Build();
                    config.AddAzureAppConfiguration(options =>
                    {
                        options.Connect(settings["ConnectionStrings:AppConfig"])
                            .ConfigureKeyVault(kv => { kv.SetCredential(new DefaultAzureCredential()); });
                    });
                }).UseStartup<Startup>());
}

I verified my logged in user through PowerShell to ensure it is correct:

And these are the role assignments in Access Control:

I know the error references "observe propagation time", but it's been hours and the same error occurred yesterday. Is this a tenant issue? Azure KV or RBAC configuration issue? Code issue? Dev environment/user account issue?

Answer 1

Being an Owner or contributor does not give you access to read keys from the key vault.

Being owner does give you the right to grant yourself access to read the keys.

If you give yourself the key vault administrator role you will be able to read the keys.

Key Vault Contributor role is for management plane operations to manage key vaults. It does not allow access to keys, secrets and certificates.

See: https://learn.microsoft.com/en-us/azure/key-vault/general/rbac-guide?tabs=azure-cli