CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-dataflowgoogle-cloud-pubsub
tombrawz
tombrawz

Reputation: 137

Dataflow job throws error when publishing to Pub/Sub topic

I have streaming Dataflow job that sink the output data to a pub/sub topic. But randomly the job logs throws an error:

There were errors attempting to publish to topic projects/my_project/topics/my_topics. Recent publish calls completed successfully: 574, recent publish calls completed with error: 1.

there is no stack trace provided by the dataflow, and from the job metrics the error type is "unavailable". After some time, the error will stopped and the pipeline still running as usual. Does this error occurs because of internal error in the GCP service, or because of quota issue? The output request was peaked at 10 req/s.

Upvotes: 4

Views: 1355

Answers (2)

no-stale-reads
no-stale-reads

Reputation: 358

Well, in my case I found out that I had an issue with the schema I was sending. In my situation, the PubSub topic was sinking to BigQuery and the tables column was expecting an int although I was sending a float.

Upvotes: 0

pavneet kohli
pavneet kohli

Reputation: 1

Found the similar issue, it resolved by adding the "Pub/Sub Admin" permission to Compute Engine default service account under IAM permissions.

Upvotes: -3

Related Questions