CODEWITHSUNDEEP
google-cloud-platformgcloudgoogle-cloud-pubsubstackdrivergoogle-cloud-stackdriver
Matt Weiden
Matt Weiden

Reputation: 13

How can I create a pubsub log sink to a different google cloud project/org?

I would like to define the steps in making a pubsub log export sink from one project to another. As a secondary goal, I would like the sink to bridge organizations. So far I have followed these steps as outlined in the gcloud help pages and the auth documentation.

First I created a sink:

# from project A
gcloud logging sinks create \
    <sink_name> \
    pubsub.googleapis.com/projects/<project_B>/topics/<topic_name> \
    --log-filter <filter>

The CLI returns successfully and gives a little advice on setting up permissions for a service account it created:

Created [https://logging.googleapis.com/v2/projects/<project_A>/sinks/<sink_name>].
Please remember to grant `serviceAccount:<new_user>@logging-<project_A_account_number>.iam.gserviceaccount.com` Pub/Sub Publisher role to the topic.
More information about sinks can be found at https://cloud.google.com/logging/docs/export/configure_export

Following this advice, I gave the new service account the appropriate permissions for that topic.

gcloud projects add-iam-policy-binding <project_B> \
    --member serviceAccount:<new_user>@logging-<project_A_account_numbe_id>.iam.gserviceaccount.com \
    --role roles/pubsub.publisher

This command returns without issue.

In spite of everything seeming OK, no logs flow through the sink.

Here are some clues: The Exports tab on the Logs Viewer reports a permissions error in the sink. The Project Activity tab reports a permissions issue.

Image: Logs Viewer, Exports

Image: Project, Activity

Is there a solution to make this work? Is it possible to generalize this to send logs to a sink in this project from other gcloud organizations?

Upvotes: 1

Views: 4174

Answers (2)

Summit Raj
Summit Raj

Reputation: 880

Matt, your gcloud commands to create a sink and grant publisher role to the service account used by the sink looks correct. The error you saw may be transient and it should have resolved after some time. A possible reason for the error could be that there was a time delay between the two commands, and the sink tried to export logs immediately before you granted the IAM role. Can you confirm that the error resolved itself eventually?

To answer your last question, yes you can export logs from one project to a destination in another project in a different organization.

Upvotes: 0

dsesto
dsesto

Reputation: 8178

I have been able to reproduce the scenario you wanted. Let me set the basics for the scenario description:

  • Project A (where my logs are stored): project-a
  • Project B (where my Pub/Sub Topic and Subscription are): project-b
  • Topic: projects/project-b/topics/myTopic
  • Subscription: mySub
  • Sink: test-sink

Then, this is the processed I followed:

  1. In project A: create a filter for logs in the Logging > Logs tab in the Google Cloud Console.

  2. Create an export with the elements in the image below (remember to append pubsub.googleapis.com/ to the name of your topic in the other project): enter image description here

  3. Move to the Exports tab and copy the Writer Identity, which should have the format [email protected]

  4. In project B: go to the IAM & admin > IAM tab in the Console and add a new member being the previous service account obtained in step 3 with the role Pub/Sub Editor enabled.

  5. Create a Pub/Sub subscription with the command gcloud beta pubsub subscriptions create --topic myTopic mySub

  6. Do some operation that results in logs read by the filter you specified in Project A.

  7. Consume the logs written to the topic using the subscription, with the command gcloud beta pubsub subscriptions pull mySub.

There you will find the logs that are written from Project A to your tropic in Project B. I have reproduced this same scenario writing logs from a simple App Engine application (and therefore with the appropriate log filter searching for App Engine logs), and when I make requests to the App Engine app, some logs are created and then written in myTopic, which I can read using mySub.

Regarding your second question, I cannot make sure whether this same procedure works in a cross-organization scenario but I see no issues with that.

Upvotes: 2

Related Questions