CODEWITHSUNDEEP
goapache-kafkaapache-beamapache-beam-io
Daisy W
Daisy W

Reputation: 21

KafkaIO Connector/Apache Beam Transform "go" SDK Available?

I am working on building a data ingestion pipeline using Apache Beam "go" SDK.

My pipeline is to consume data from Kafka queue and persist the data to Google Cloud Bigtable (and/or to another Kafka topic).

So far, I have not been able to find a Kafka IO Connector (also known as Apache I/O Transform) written in "go" (I was able to find a java version, however).

Here's link to supported Apache Beam built-in I/O transforms: https://beam.apache.org/documentation/io/built-in/

I am looking for the "go" equivalent of the following Java code:

    pipeline.apply("kafka_deserialization", KafkaIO.<String, String>read()
        .withBootstrapServers(KAFKA_BROKER)
        .withTopic(KAFKA_TOPIC)
        .withConsumerConfigUpdates(CONSUMER_CONFIG)
        .withKeyDeserializer(StringDeserializer.class)
        .withValueDeserializer(StringDeserializer.class))

Do you have any information on the availability of KafkaIO Connector "go" SDK/library?

Upvotes: 1

Views: 557

Answers (1)

Daisy W
Daisy W

Reputation: 21

@cricket_007 In case you are also curious, I received the following update from Robert Burke ([email protected]) who is in the Apache Beam team:

There presently isn't a Kafka transform for Go. 

The Go SDK is still experimental, largely due to scalable IO support, which is why the Go SDK isn't represented in the built-in io page.

There's presently no way for an SDK user to write a Streaming source in the Go SDK, since there's no mechanism for a DoFn to "self terminate" bundles, such as to allow for scalability and windowing from streaming sources. 

However, SplittableDoFns are on their way, and will eventually be the solution for writing these.

At present, the Beam Go SDK IOs haven't been tested and vetted for production use. Until the initial SplittableDoFn support is added to the Go SDK, Batch transforms cannot split, and can't scale beyond a single worker thread. This batch version should land in the next few months, and the streaming version land a few months after that, after which a Kafka IO can be developed. 

I wish I had better news for you, but I can say progress is being made.

Robert Burke

Upvotes: 1

Related Questions