google-cloud-platformterraformgoogle-cloud-pubsubterraform-provider-gcp

Reputation: 100

Terraform gcloud

I have the following setup in Google Cloud:

application 'generator' which publishes messages to a Google Cloud PubSub topic.
application 'worker' which consumes a unique message.
any invalid PubSub messages should end up in a 'dead letter' topic.

This topic should have a 'dead letter' topic where invalid messages end up. However, whenever I configure this via Terraform, the google cloud console mentions I do not have the 'subscriber' and 'publisher' role attached to my project pubsub service account:

I have the following terraform configuration which seems to be correct AFAIK:

resource "google_project_service_identity" "pubsub_sa" {
    provider = google-beta
    
    project = var.project_id
    service = "pubsub.googleapis.com"
}

/* ... topic and dead-letter topic config here ... */

data "google_iam_policy" "project_pubsub_publishers" {
    binding {
        role = "roles/pubsub.publisher"
        members = [
            "serviceAccount:${google_service_account.project_generator_serviceaccount.email}",
            "serviceAccount:${google_service_account.project_worker_serviceaccount.email}",
            "serviceAccount:${google_project_service_identity.pubsub_sa.email}",
        ]
    }
}

resource "google_pubsub_topic_iam_policy" "project_request_publishers" {
    project  = var.project_id
    topic = google_pubsub_topic.generator_request_pubsub.name
    policy_data = data.google_iam_policy.project_pubsub_publishers.policy_data
}

data "google_iam_policy" "project_pubsub_subscribers" {
    binding {
        role = "roles/pubsub.subscriber"
        members = [
            "serviceAccount:${google_service_account.project_generator_serviceaccount.email}",
            "serviceAccount:${google_service_account.project_worker_serviceaccount.email}",
            "serviceAccount:${google_project_service_identity.pubsub_sa.email}",
        ]
    }
}

resource "google_pubsub_topic_iam_policy" "project_request_subscribers" {
    topic = google_pubsub_topic.generator_request_pubsub.name
    project  = var.project_id
    policy_data = data.google_iam_policy.project_pubsub_subscribers.policy_data
}

Clicking 'Add' in the web gui and then doing a terraform plan shows following changes:

Terraform will perform the following actions:

  # module.gcloud.google_pubsub_topic_iam_policy.project_invalid_request_publishers will be updated in-place
  ~ resource "google_pubsub_topic_iam_policy" "project_invalid_request_publishers" {
        id          = "projects/MY-GCLOUD-PROJECTID/topics/generator-request-pubsub-invalid"
      ~ policy_data = jsonencode(
          ~ {
              ~ bindings = [
                  ~ {
                      ~ members = [
                          + "serviceAccount:[email protected]",
                          + "serviceAccount:[email protected]",
                            "serviceAccount:[email protected]",
                        ]
                        # (1 unchanged element hidden)
                    },
                  - {
                      - members = [
                          - "serviceAccount:[email protected]",
                          - "serviceAccount:[email protected]",
                          - "serviceAccount:[email protected]",
                        ]
                      - role    = "roles/pubsub.subscriber"
                    },
                ]
            }
        )
        # (3 unchanged attributes hidden)
    }

  # module.gcloud.google_pubsub_topic_iam_policy.project_invalid_request_subscribers will be updated in-place
  ~ resource "google_pubsub_topic_iam_policy" "project_invalid_request_subscribers" {
        id          = "projects/MY-GCLOUD-PROJECTID/topics/generator-request-pubsub-invalid"
      ~ policy_data = jsonencode(
          ~ {
              ~ bindings = [
                  - {
                      - members = [
                          - "serviceAccount:[email protected]",
                        ]
                      - role    = "roles/pubsub.publisher"
                    },
                    {
                        members = [
                            "serviceAccount:[email protected]",
                            "serviceAccount:[email protected]",
                            "serviceAccount:[email protected]",
                        ]
                        role    = "roles/pubsub.subscriber"
                    },
                ]
            }
        )
        # (3 unchanged attributes hidden)
    }

  # module.gcloud.google_pubsub_topic_iam_policy.project_request_subscribers will be updated in-place
  ~ resource "google_pubsub_topic_iam_policy" "project_request_subscribers" {
        id          = "projects/MY-GCLOUD-PROJECTID/topics/generator-request-pubsub"
      ~ policy_data = jsonencode(
          ~ {
              ~ bindings = [
                  ~ {
                      ~ role    = "roles/pubsub.publisher" -> "roles/pubsub.subscriber"
                        # (1 unchanged element hidden)
                    },
                ]
            }
        )
        # (3 unchanged attributes hidden)
    }

But I'm not sure what I'm doing wrong here. Any ideas?

Upvotes: 3

Answers (3)

SRJ

Reputation: 2816

I have fixed this problem using below simple terraform configuration.

Load google project configuration

data "google_project" "project" {
  project_id = var.gcp_project
}

Declare PubSub Topic IAM binding resource to attach to a topic

resource "google_pubsub_topic_iam_binding" "publisher" {
  project = var.project

  role    = "roles/pubsub.publisher"
  topic   = google_pubsub_topic.deadletter.id
  members = ["serviceAccount:service-${data.google_project.project.number}@gcp-sa-pubsub.iam.gserviceaccount.com"]
}

Declare PubSub Subscription IAM binding resource to attach to a subscription


resource "google_pubsub_subscription_iam_binding" "subscriber" {
  project = var.project

  role         = "roles/pubsub.subscriber"
  subscription = google_pubsub_subscription.this.id
  members      = ["serviceAccount:service-${data.google_project.project.number}@gcp-sa-pubsub.iam.gserviceaccount.com"]
}

Note : I didn't use `google_pubsub_topic_iam_policy` and `google_iam_policy`.

Upvotes: 0

AudronFS

Reputation: 149

Jaime is right, you need to add those IAM policies to

"service-${project-number}@gcp-sa-pubsub.iam.gserviceaccount.com"

It is a specific sa hidden from the main ones. You can find it in the console in >IAM and select the check box on the top right corner "include google-provided role grants"

There is also needed to add a google_pubsub_topic_iam_policy.

Here is a Terraform working example

data "google_project" "current" {}

data "google_iam_policy" "publisher" {
  binding {
    role = "roles/pubsub.publisher"
    members = [
      "serviceAccount:service-${data.google_project.current.number}@gcp-sa-pubsub.iam.gserviceaccount.com",
    ]
  }
}
resource "google_pubsub_topic_iam_policy" "policy" {
  project = var.project
  topic = google_pubsub_topic.yourTopic.name
  policy_data = data.google_iam_policy.publisher.policy_data
}
data "google_iam_policy" "subscriber" {
  binding {
    role = "roles/pubsub.subscriber"
    members = [
      "serviceAccount:service-${data.google_project.current.number}@gcp-sa-pubsub.iam.gserviceaccount.com",
    ]
  }
}
resource "google_pubsub_subscription_iam_policy" "policy" {
  subscription = google_pubsub_subscription.yourSubscription.name
  policy_data  = data.google_iam_policy.subscriber.policy_data
}

Upvotes: 3

Jaime López

Reputation: 466

As per the documentation, seems that you need to first actually set the configuration for a 'dead-letter topic' in GCP.

Setting a dead-letter topic

Which (among some other information) states that:

To create a subscription and set a dead-letter topic, use the gcloud pubsub subscriptions create command:

gcloud pubsub subscriptions create subscription-id \
  --topic=topic-id \
  --dead-letter-topic=dead-letter-topic-id \
  [--max-delivery-attempts=max-delivery-attempts] \
  [--dead-letter-topic-project=dead-letter-topic-project]

To update a subscription and set a dead-letter topic, use the gcloud pubsub subscriptions update command:

gcloud pubsub subscriptions update subscription-id \
  --dead-letter-topic=dead-letter-topic-id \
  [--max-delivery-attempts=max-delivery-attempts] \
  [--dead-letter-topic-project=dead-letter-topic-project]

Granting forwarding permissions
To forward undeliverable messages to a dead-letter topic, Pub/Sub must have permission to do the following:

Publish messages to the topic.
Acknowledge the messages, which removes them from the subscription.

Pub/Sub creates and maintains a service account for each project: [email protected]. You can grant forwarding permissions by assigning publisher and subscriber roles to this service account. If you configured the subscription using Cloud Console, the roles are granted automatically.

Assigning Pub/Sub the publisher role
To grant Pub/Sub permission to publish messages to a dead-letter topic, run the following command:

PUBSUB_SERVICE_ACCOUNT="service-${project-number}@gcp-sa-pubsub.iam.gserviceaccount.com"

gcloud pubsub topics add-iam-policy-binding dead-letter-topic-id \
    --member="serviceAccount:$PUBSUB_SERVICE_ACCOUNT"\
    --role="roles/pubsub.publisher"

Assigning Pub/Sub the subscriber role
To grant Pub/Sub permission to acknowledge forwarded undeliverable messages, run the following command:

PUBSUB_SERVICE_ACCOUNT="service-${project-number}@gcp-sa-pubsub.iam.gserviceaccount.com"

gcloud pubsub subscriptions add-iam-policy-binding subscription-id \
    --member="serviceAccount:$PUBSUB_SERVICE_ACCOUNT"\
    --role="roles/pubsub.subscriber"

Hope this is helpful for you. Regards.

Upvotes: 1