CODEWITHSUNDEEP
google-cloud-platformapache-beamdataflowbigtable
bya
bya

Reputation: 46

Is there an Apache Beam DynamicDestinations for Bigtable IO connector?

Apache Beam has a DynamicDestinations! for BigQueryIO.

Is there any same implementations or workaround for BigtableIO? 

events.apply(BigTableIO.<UserEvent>write()
  .to(new DynamicDestinations<UserEvent, String>() {
...
...

Upvotes: 0

Views: 127

Answers (1)

guillaume blaquiere
guillaume blaquiere

Reputation: 75920

I would say no because the load mode in BigQuery and BigTable is different.

Indeed, with BigQuery, you can import chunk of data, grouped in files, and load them in different table. The dynamic destination perform exactly this: the data are routed in the correct chunk and then loaded into BigQuery.

However, for BigTable, this kind of load doesn't exist. Thus the same logic can't be applied.

As workaround, you can write yourself into BigTable according with your data values or write a file and have another Dataflow to load data from Storage to BigTable (Dataflow template exists for this)

Upvotes: 2

Related Questions