io.grpc.StatusRuntimeException: PERMISSION_DENIED: User not authorized to perform this action

Question

I try to implement simple example of spring cloud config + spring cloud bus.

So I've implemented

1. Client server application
2. Config server application

and in the separated repository I've put application.properties file.

Looks like everything is correct on the server side. But on client side I see error when application is starting:

org.springframework.cloud.stream.binder.BinderException: Exception thrown while building outbound endpoint
    at org.springframework.cloud.stream.binder.AbstractMessageChannelBinder.doBindProducer(AbstractMessageChannelBinder.java:192) ~[spring-cloud-stream-2.1.0.RELEASE.jar:2.1.0.RELEASE]
    at org.springframework.cloud.stream.binder.AbstractMessageChannelBinder.doBindProducer(AbstractMessageChannelBinder.java:93) ~[spring-cloud-stream-2.1.0.RELEASE.jar:2.1.0.RELEASE]
    at org.springframework.cloud.stream.binder.AbstractBinder.bindProducer(AbstractBinder.java:139) ~[spring-cloud-stream-2.1.0.RELEASE.jar:2.1.0.RELEASE]
    at org.springframework.cloud.stream.binding.BindingService.lambda$rescheduleProducerBinding$2(BindingService.java:267) ~[spring-cloud-stream-2.1.0.RELEASE.jar:2.1.0.RELEASE]
    at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-5.1.2.RELEASE.jar:5.1.2.RELEASE]
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[na:na]
    at java.base/java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264) ~[na:na]
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java) ~[na:na]
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
    at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]
Caused by: com.google.api.gax.rpc.PermissionDeniedException: io.grpc.StatusRuntimeException: PERMISSION_DENIED: User not authorized to perform this action.
    at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:55) ~[gax-1.49.1.jar:1.49.1]
    at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:72) ~[gax-grpc-1.49.1.jar:1.49.1]
    at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:60) ~[gax-grpc-1.49.1.jar:1.49.1]
    at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97) ~[gax-grpc-1.49.1.jar:1.49.1]
    at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68) ~[api-common-1.8.1.jar:na]
    at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:982) ~[guava-28.1-android.jar:na]
    at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) ~[guava-28.1-android.jar:na]
    at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1138) ~[guava-28.1-android.jar:na]
    at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:957) ~[guava-28.1-android.jar:na]
    at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:748) ~[guava-28.1-android.jar:na]
    at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:515) ~[grpc-stub-1.24.1.jar:1.24.1]
    at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:490) ~[grpc-stub-1.24.1.jar:1.24.1]
    at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) ~[grpc-api-1.24.1.jar:1.24.1]
    at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) ~[grpc-api-1.24.1.jar:1.24.1]
    at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) ~[grpc-api-1.24.1.jar:1.24.1]
    at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:700) ~[grpc-core-1.24.1.jar:1.24.1]
    at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) ~[grpc-api-1.24.1.jar:1.24.1]
    at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) ~[grpc-api-1.24.1.jar:1.24.1]
    at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) ~[grpc-api-1.24.1.jar:1.24.1]
    at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:399) ~[grpc-core-1.24.1.jar:1.24.1]
    at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:510) ~[grpc-core-1.24.1.jar:1.24.1]
    at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:66) ~[grpc-core-1.24.1.jar:1.24.1]
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:630) ~[grpc-core-1.24.1.jar:1.24.1]
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:518) ~[grpc-core-1.24.1.jar:1.24.1]
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:692) ~[grpc-core-1.24.1.jar:1.24.1]
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:681) ~[grpc-core-1.24.1.jar:1.24.1]
    at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.24.1.jar:1.24.1]
    at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) ~[grpc-core-1.24.1.jar:1.24.1]
    ... 7 common frames omitted
    Suppressed: com.google.api.gax.rpc.AsyncTaskException: Asynchronous task failed
        at com.google.api.gax.rpc.ApiExceptions.callAndTranslateApiException(ApiExceptions.java:57) ~[gax-1.49.1.jar:1.49.1]
        at com.google.api.gax.rpc.UnaryCallable.call(UnaryCallable.java:112) ~[gax-1.49.1.jar:1.49.1]
        at com.google.cloud.pubsub.v1.TopicAdminClient.createTopic(TopicAdminClient.java:256) ~[google-cloud-pubsub-1.101.0.jar:1.101.0]
        at com.google.cloud.pubsub.v1.TopicAdminClient.createTopic(TopicAdminClient.java:203) ~[google-cloud-pubsub-1.101.0.jar:1.101.0]
        at org.springframework.cloud.gcp.pubsub.PubSubAdmin.createTopic(PubSubAdmin.java:107) ~[spring-cloud-gcp-pubsub-1.2.1.RELEASE.jar:1.2.1.RELEASE]
        at org.springframework.cloud.gcp.stream.binder.pubsub.provisioning.PubSubChannelProvisioner.makeSureTopicExists(PubSubChannelProvisioner.java:121) ~[spring-cloud-gcp-pubsub-stream-binder-1.2.1.RELEASE.jar:1.2.1.RELEASE]
        at org.springframework.cloud.gcp.stream.binder.pubsub.provisioning.PubSubChannelProvisioner.provisionProducerDestination(PubSubChannelProvisioner.java:63) ~[spring-cloud-gcp-pubsub-stream-binder-1.2.1.RELEASE.jar:1.2.1.RELEASE]
        at org.springframework.cloud.gcp.stream.binder.pubsub.provisioning.PubSubChannelProvisioner.provisionProducerDestination(PubSubChannelProvisioner.java:45) ~[spring-cloud-gcp-pubsub-stream-binder-1.2.1.RELEASE.jar:1.2.1.RELEASE]
        at org.springframework.cloud.stream.binder.AbstractMessageChannelBinder.doBindProducer(AbstractMessageChannelBinder.java:174) ~[spring-cloud-stream-2.1.0.RELEASE.jar:2.1.0.RELEASE]
        at org.springframework.cloud.stream.binder.AbstractMessageChannelBinder.doBindProducer(AbstractMessageChannelBinder.java:93) ~[spring-cloud-stream-2.1.0.RELEASE.jar:2.1.0.RELEASE]
        at org.springframework.cloud.stream.binder.AbstractBinder.bindProducer(AbstractBinder.java:139) ~[spring-cloud-stream-2.1.0.RELEASE.jar:2.1.0.RELEASE]
        at org.springframework.cloud.stream.binding.BindingService.lambda$rescheduleProducerBinding$2(BindingService.java:267) ~[spring-cloud-stream-2.1.0.RELEASE.jar:2.1.0.RELEASE]
        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-5.1.2.RELEASE.jar:5.1.2.RELEASE]
        ... 7 common frames omitted
Caused by: io.grpc.StatusRuntimeException: PERMISSION_DENIED: User not authorized to perform this action.
    at io.grpc.Status.asRuntimeException(Status.java:533) ~[grpc-api-1.24.1.jar:1.24.1]
    ... 24 common frames omitted

Based on stacktrace I thought that problem related with permissions. And to solve it I've granted owner permissions:

But it didn't help for some reason.

Let me share my code base:

application.properties file is empty.

build.gradle.kts:

plugins {
// For upwards compatibility check
    val springBootVersion = "2.1.0.RELEASE"
    id("org.springframework.boot") version springBootVersion
    ...
}

apply(plugin = "io.spring.dependency-management")
...

group = "my.gcp.samples"
version = "0.0.1-SNAPSHOT"

val developmentOnly = configurations.create("developmentOnly")
configurations {
    developmentOnly
    runtimeClasspath {
        extendsFrom(developmentOnly)
    }
    compileOnly {
        extendsFrom(configurations.annotationProcessor.get())
    }
}

java {
    sourceCompatibility = JavaVersion.VERSION_11
}
repositories {
    mavenCentral()
// Milestone spring repository
    maven(url = "https://repo.spring.io/milestone/")
}

val springCloudVersion = "Greenwich.RELEASE"
val testcontainersVersion = "1.11.4"
val junitVersion = "5.5.1"

dependencies {
    // GCP bucket
    implementation("org.springframework.boot", "spring-boot-starter-web")
    implementation("org.springframework.cloud", "spring-cloud-gcp-starter")
    implementation("org.springframework.cloud", "spring-cloud-gcp-starter-storage")
    // GCP big query
    implementation("org.springframework.cloud", "spring-cloud-gcp-starter-bigquery")
    implementation("org.springframework.integration", "spring-integration-core")
    implementation("org.springframework.boot", "spring-boot-starter-thymeleaf")
    //cloud bus
    implementation ("org.springframework.cloud:spring-cloud-gcp-starter-bus-pubsub")
    implementation ("org.springframework.cloud:spring-cloud-config-client")
}

configure<DependencyManagementExtension> {
    imports {
        mavenBom("org.springframework.cloud:spring-cloud-dependencies:$springCloudVersion")
        mavenBom("org.springframework.cloud:spring-cloud-gcp-dependencies:1.2.1.RELEASE")
        mavenBom("org.junit:junit-bom:$junitVersion")
    }
}

Application:

@SpringBootApplication
public class Application {

    public static void main(String... args) {
        SpringApplication.run(Application.class, args);
    }

}

ExampleController:

@RefreshScope
@RestController
public class ExampleController {

    @Value("${example.message:none}")
    private String message;

    @GetMapping("/message")
    public String getMessage() {
        return this.message;
    }

}

I would like to say that application works partially despite the exception thrown on startup. When I make a GET localhost:8080/message I see value from file located at the github repository. But if I change this file in the repository then GET `localhost:8080/message returns old value.

What wrong with my code ? How to fix it ?

Answer 1

Sure, firstly, you need to check permissions for your service account in Cloud Console.

In my case, it also required the correct project ID in "spring.cloud.gcp.project-id" property in my spring configuration.

Answer 2

In my case I had missed adding pubsub editor role to my topic. After adding it solved issue for me. Take a close look at permission tab and add necessary pincipals.

Answer 3

I've got the same exception when there was a change in the underlying topic. It worked after I deleted the subscription and created it again with the new topic and gave Subscriber permission in the cloud console page for the service account and then error disappeared.

Answer 4

I notice in the error message that you need the access to create a new topic:

Suppressed: com.google.api.gax.rpc.AsyncTaskException: Asynchronous task failed
        at com.google.api.gax.rpc.ApiExceptions.callAndTranslateApiException(ApiExceptions.java:57) ~[gax-1.49.1.jar:1.49.1]
        at com.google.api.gax.rpc.UnaryCallable.call(UnaryCallable.java:112) ~[gax-1.49.1.jar:1.49.1]
        at com.google.cloud.pubsub.v1.TopicAdminClient.createTopic(TopicAdminClient.java:256) ~[google-cloud-pubsub-1.101.0.jar:1.101.0]
        at com.google.cloud.pubsub.v1.TopicAdminClient.createTopic(TopicAdminClient.java:203) ~[google-cloud-pubsub-1.101.0.jar:1.101.0]
        at org.springframework.cloud.gcp.pubsub.PubSubAdmin.createTopic(PubSubAdmin.java:107) ~[spring-cloud-gcp-pubsub-1.2.1.RELEASE.jar:1.2.1.RELEASE]
        at org.springframework.cloud.gcp.stream.binder.pubsub.provisioning.PubSubChannelProvisioner.makeSureTopicExists(PubSubChannelProvisioner.java:121) ~[spring-cloud-gcp-pubsub-stream-binder-1.2.1.RELEASE.jar:1.2.1.RELEASE]
        at org.springframework.cloud.gcp.stream.binder.pubsub.provisioning.PubSubChannelProvisioner.provisionProducerDestination(PubSubChannelProvisioner.java:63) ~[spring-cloud-gcp-pubsub-stream-binder-1.2.1.RELEASE.jar:1.2.1.RELEASE]
        at org.springframework.cloud.gcp.stream.binder.pubsub.provisioning.PubSubChannelProvisioner.provisionProducerDestination(PubSubChannelProvisioner.java:45) ~[spring-cloud-gcp-pubsub-stream-binder-1.2.1.RELEASE.jar:1.2.1.RELEASE]
        at org.springframework.cloud.stream.binder.AbstractMessageChannelBinder.doBindProducer(AbstractMessageChannelBinder.java:174) ~[spring-cloud-stream-2.1.0.RELEASE.jar:2.1.0.RELEASE]
        at org.springframework.cloud.stream.binder.AbstractMessageChannelBinder.doBindProducer(AbstractMessageChannelBinder.java:93) ~[spring-cloud-stream-2.1.0.RELEASE.jar:2.1.0.RELEASE]
        at org.springframework.cloud.stream.binder.AbstractBinder.bindProducer(AbstractBinder.java:139) ~[spring-cloud-stream-2.1.0.RELEASE.jar:2.1.0.RELEASE]
        at org.springframework.cloud.stream.binding.BindingService.lambda$rescheduleProducerBinding$2(BindingService.java:267) ~[spring-cloud-stream-2.1.0.RELEASE.jar:2.1.0.RELEASE]
        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-5.1.2.RELEASE.jar:5.1.2.RELEASE]
        ... 7 common frames omitted
Caused by: io.grpc.StatusRuntimeException: PERMISSION_DENIED: User not authorized to perform this action

First, I'd like to clarify:

1. How you pass your credentials? The following snippets will show how you can set a service account authentication:

ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(<Service-account-as-string>.getBytes());
CredentialsProvider credentialsProvider =
      FixedCredentialsProvider.create(
          ServiceAccountCredentials.fromStream(byteArrayInputStream));


TopicAdminSettings.Builder topicAdminBuilder = TopicAdminSettings.newBuilder();
if (credentialsProvider != null) {
  topicAdminBuilder.setCredentialsProvider(credentialsProvider);
}
TopicAdminClient topicAdminClient = TopicAdminClient.create(topicAdminBuilder.build());

2. If you're using service account, you can check whether your service account is allowed to create a new topic.

Hope it helps.