CODEWITHSUNDEEP
google-cloud-datastoregoogle-cloud-dataflow
Gilmar Souza Jr.
Gilmar Souza Jr.

Reputation: 65

Querying namespaces using Dataflow's DatastoreIO

Is it possible to query entities in a specific namespace when using Dataflow's DatastoreIO?

Upvotes: 3

Views: 248

Answers (1)

jkff
jkff

Reputation: 17913

As of today, unfortunately no - DatastoreIO does not support reading from entities in namespaces due to limitations of the Datastore QuerySplitter API which is used to read results of a query in parallel. We are tracking the issue internally and your feedback is valuable for prioritizing it.

If the number of entities your pipeline reads from Datastore is small enough (or the rest of the processing heavy enough) that reading them sequentially (but processing in parallel) would be ok, you can try the workaround suggested in Google Cloud Dataflow User-Defined MySQL Source

You can also try exporting your data to BigQuery and processing it there, using BigQuery's querying capabilities or Dataflow's BigQueryIO connectors - those have no parallelism limitations.

Upvotes: 3

Related Questions