CODEWITHSUNDEEP
google-cloud-platformgoogle-bigquerygoogle-cloud-pubsub
marian.vladoi
marian.vladoi

Reputation: 8074

Error trying to create a GCP Pub Sub BigQuery subscriptions

I am trying to create a GCP Pub Sub BigQuery subscriptions using the console: https://cloud.google.com/pubsub/docs/bigquery

However, I get the following error message:

API returned error: ‘Request contains an invalid argument.’

Any help would be appreciated.

enter image description here

enter image description here

NOTE

  1. When the big query table does not exist I get the following error: enter image description here

2.Pub Sub schema was not deleted enter image description here

Upvotes: 3

Views: 2763

Answers (2)

Cleverson Zocche Sato
Cleverson Zocche Sato

Reputation: 71

Update: Actually, despite the generic error message, turns out to be a straightforward issue.

To use the "write metadata" option the BigQuery table must previously have the following field structure:

  • subscription_name (string),
  • message_id (string)
  • publish_time (timestamp)
  • data (bytes, string or json)
  • attributes (string or json)

they are better described in the docs here

Once they are created, this option works fine.

I believe that problem with the "Use topic schema" is also related to the schema used in the topic, since the table must have already the same structure (but there is a need to check it in your configuration). If your topic follow an avro schema, this might help: https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-avro#avro_conversions

---------- previous answer

Not a definitive answer, but am having the same problem and figured out that it is somewhat related to the following options:

Use topic schema

Write metadata

Unchecking then makes it work.

The same thing happens algo using terraform to try to set up the infrastructure.

I am still investigating if it is a bug or perhaps an error in my schema definition, but maybe it can be a starting point for you as well

Upvotes: 2

Kamal Aboul-Hosn
Kamal Aboul-Hosn

Reputation: 17261

A 404 error would indicate that either the BigQuery table does not exist or the Pub/Sub schema associated with the topic was deleted. For the former, ensure that the project, data set, and table names all match the names of the existing table to which you want to write data. For the latter, you could look at the topic details page and make sure that the schema name is not _deleted-schema_.

Upvotes: 0

Related Questions