CODEWITHSUNDEEP
google-bigquery
foxwendy
foxwendy

Reputation: 2929

Bigquery: How long should we wait to stream data after schema update?

I add three columns into my existing tables (change table schema). I can see them on the web UI. But when I stream the data in , I have errors: no such field. I printed out the data content json, found it actually matched the schema.

How long should I wait for the changed schema to be seen for streaming? Any rule of thumb?

Upvotes: 1

Views: 524

Answers (1)

Jordan Tigani
Jordan Tigani

Reputation: 26627

The cache invalidation frequency is currently set to 8 hours for streaming schemas. So if you change your schema, it may take up to 8 hours to be able to stream data to that table.

This is set so high in order to deal with very high rate of inserts without overwhelming our metadata servers.

We've got an internal bug to lower the cache time to a few minutes or less.

Upvotes: 1

Related Questions