CODEWITHSUNDEEP
google-cloud-platformgoogle-bigqueryavrogoogle-cloud-pubsub
Jonas Laux
Jonas Laux

Reputation: 449

Creating a BigQuery PubSub Subscription with type JSON fields

I'm currently creating a BigQuery table, which has the experimental type JSON with Terraform. I want to insert data through a BigQuery PubSub Subscription. The PubSub Subscription's Topic schema is written in AVRO. I was following this documentation to map the fields, though because JSON is still experimental for BigQuery, i couldn't find any docu for that. When i try to apply my code, i get:

Error: Error creating Subscription: googleapi: Error 400: Incompatible schema type for field data: STRING vs. JSON Details: [   {
    "@type": "type.googleapis.com/google.rpc.ErrorInfo",
    "domain": "pubsub.googleapis.com",
    "metadata": {
      "actual_value": "JSON",
      "expected_value": "STRING",
      "field_name": "data",
      "reason": "INCOMPATIBLE_TYPE"
    },
    "reason": "INCOMPATIBLE_SCHEMA"   } ]

My BigQuery Schema (part) looks like this:

  {
    "mode": "NULLABLE",
    "name": "data",
    "type": "JSON",
    "description": "Content of the event"
  }

My PubSub Topic AVRO Schema (part) looks like this:

  {
    "name": "data",
    "type": ["null", { "type": "string", "sqlType": "JSON" }],
    "doc": "Content of the event",
    "default": null
  },

Upvotes: 1

Views: 1970

Answers (2)

Kamal Aboul-Hosn
Kamal Aboul-Hosn

Reputation: 17281

Update: BigQuery subscriptions support the JSON type as of 11/15/2022.

Original answer: Pub/Sub BigQuery subscriptions do not yet support the JSON type, though support for it is coming within the next few weeks. When that happens, the schemas you have should be considered compatible. Right now, the only way to get the BigQuery schema and the Pub/Sub schema to be compatible is to use the STRING type in both.

Upvotes: 4

robertsahlin
robertsahlin

Reputation: 541

It is already supported as of 11 November 2022, however not officially announced/released yet. You can read more at https://robertsahlin.substack.com/p/level-up-your-bigquery-pubsub-subscriptions?sd=pf

Upvotes: 5

Related Questions