CODEWITHSUNDEEP
javagoogle-cloud-platformgoogle-bigquery
vpram86
vpram86

Reputation: 6000

java bigquery storage write api

Using the java bigquery storage api as documented here https://cloud.google.com/bigquery/docs/write-api. Keeping the write stream long lived and refreshing it when one of the non-retry-able errors happened as per this https://cloud.google.com/bigquery/docs/write-api#error_handling

I am sticking with default stream. I have two tables and different parts of code responsible for writing to each table, maintaining its own stream writer.

If data is flowing, everything is fine. No errors. However I want to test refreshing the stream writers work too so I wait for default stream timeout (10mins) which closes the stream and try writing again. I can create the stream fine, no error there, but for one of the table I keep getting cancelled error wrapped in a Pre condition failed making my code refresh again and again.

Original error because stream closed due to inactivity

! io.grpc.StatusRuntimeException: FAILED_PRECONDITION: Stream is closed due to com.google.api.gax.rpc.AbortedException: io.grpc.StatusRuntimeException: ABORTED: Closing the stream because it has been inactive for 600 seconds. Entity: projects/<id>/datasets/<id>/tables/<id>/_default
! at com.google.cloud.bigquery.storage.v1beta2.StreamWriterV2.appendInternal(StreamWriterV2.java:263)
! at com.google.cloud.bigquery.storage.v1beta2.StreamWriterV2.append(StreamWriterV2.java:234)
! at com.google.cloud.bigquery.storage.v1beta2.JsonStreamWriter.append(JsonStreamWriter.java:114)
! at com.google.cloud.bigquery.storage.v1beta2.JsonStreamWriter.append(JsonStreamWriter.java:89)

Further repeating errors on new stream(s)

! io.grpc.StatusRuntimeException: FAILED_PRECONDITION: Stream is closed due to com.google.api.gax.rpc.CancelledException: io.grpc.StatusRuntimeException: CANCELLED: io.grpc.Context was cancelled without error
! at com.google.cloud.bigquery.storage.v1beta2.StreamWriterV2.appendInternal(StreamWriterV2.java:263)
! at com.google.cloud.bigquery.storage.v1beta2.StreamWriterV2.append(StreamWriterV2.java:234)
! at com.google.cloud.bigquery.storage.v1beta2.JsonStreamWriter.append(JsonStreamWriter.java:114)
! at com.google.cloud.bigquery.storage.v1beta2.JsonStreamWriter.append(JsonStreamWriter.java:89)

I am not sure why its being cancelled without error. Any pointers on how I can debug this or recommendation on how to maintain and refresh a long-lived streaming writer?

Upvotes: 0

Views: 2471

Answers (1)

Stephanie Wang
Stephanie Wang

Reputation: 132

Updating the Java client library version should solve this problem since reconnect support was added for the JsonStreamwriter. Instead of throwing this error, it should handle retries.

Upvotes: 1

Related Questions