CODEWITHSUNDEEP
apache-beambigtablegoogle-cloud-bigtableapache-beam-io
Vasanti
Vasanti

Reputation: 1357

Is there a Apache Beam + Cloud Bigtable connector in Golang?

Is there a way to access data stored in Cloud Bigtable as the input source for running Apache Beam pipelines?

Upvotes: 2

Views: 326

Answers (1)

Robert Burke
Robert Burke

Reputation: 111

There is currently no Apache Beam Go SDK IO written for Bigtable.

One could base them on the IOs that exist (such as the BigQueryIO), but until both the Go SDK and runners support portable work spliting, there will be scaling issues, notably, serialization on the initial reads. This is in progress, but has no timeline to be ready just yet.

Another upcoming solution is "cross language IO" which will allow SDKs to use IOs from other languages (in particular, letting Python & Go use Java IOs, which have gone through productionization already).

At this time, there's been spent little to no time working on/testing the Go SDK IOs, and make no guarantees about the reliability or fit or finish of them.

Upvotes: 4

Related Questions