CODEWITHSUNDEEP
google-cloud-dataflowapache-beamapache-beam-io
user1481147
user1481147

Reputation: 1

Example of reading and writing transoforms with PubSub using apache beam python sdk

I see examples here https://cloud.google.com/dataflow/model/pubsub-io#reading-with-pubsubio for Java, but when I look here https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/pubsub.py its says:

def reader(self):
    raise NotImplementedError(
        'PubSubSource is not supported in local execution.')

What does that mean? Cloud Data Flow Python SDK PubSub Source/Sink is not quite ready?

Upvotes: 0

Views: 1308

Answers (3)

andreimarinescu
andreimarinescu

Reputation: 3641

For anyone visiting this question in 2019, I can confirm that PubSub does work with DirectRunner as long as proper Google Cloud authentication is provided.

Upvotes: 0

rezn8
rezn8

Reputation: 151

It would appear that it's not ready yet, as I was able to run it locally with the Java SDK and using the pubsub emulator, but as you've encountered, not with the Python SDK.

Upvotes: 0

jkff
jkff

Reputation: 17913

It means that reading from PubSub is currently not supported when executing the pipeline locally (on your machine, i.e. not in the cloud). Local execution is mainly used for testing.

PubSub is supported when you run using the Dataflow runner.

Upvotes: 1

Related Questions