CODEWITHSUNDEEP
google-bigquerygoogle-cloud-dataflow
Pavan Kumar Kattamuri
Pavan Kumar Kattamuri

Reputation: 335

Querying BigQuery's INFORMATION_SCHEMA.TABLES from dataflow results in error

I'm trying to run a query on INFORMATION_SCHEMA to retrieve metadata about bigquery tables from dataflow. Find the code and the error below. The same query is working fine from the BQ UI

query=SELECT * EXCEPT(is_typed) FROM `dummy_dataset.INFORMATION_SCHEMA.TABLES`"

p1=p | "create1" >> beam.io.Read(beam.io.BigQuerySource(query=query, use_standard_sql=True))

"errors": [
      {
        "message": "Illegal operation on INFORMATION_SCHEMA view: dummy_dataset.INFORMATION_SCHEMA.TABLES",
        "domain": "global",
        "reason": "invalid"
      }
    ]

Upvotes: 2

Views: 1008

Answers (1)

F10
F10

Reputation: 2893

I was able to read from the INFORMATION_SCHEMA.TABLES view from my dataset by using the runner DataflowRunner, but I got the same error message as you when I used DirectRunner. Thus, like Elliott mentions, you could file a bug for the BigQuery team, as well for the Dataflow team. In the meantime, you could try using DataflowRunner to read from this view.

Hope it helps.

Upvotes: 1

Related Questions