CODEWITHSUNDEEP
apache-beamdataflow
Encho Mishinev
Encho Mishinev

Reputation: 1

ByteStringCoder for reading data in Apache Beam

I am trying to read data with version 2.4.0 of Apache Beam using the standard TextIO.read(). The data has to be read as a ByteString.

Unfortunately, it doesn't seem like Apache Beam supports .withCoder() in the same way Dataflow does. I can't seem to find an alternative way to introduce a coder. Furthermore, it seems like ByteStringCoder is no longer included in the coders of Apache Beam.

What's the best way to get the same result of Dataflow's .withCoder(ByteStringCoder.of()) with the latest version of Apache Beam? Coders are still present in Apache Beam so there has to be some way to use them.

Upvotes: 0

Views: 296

Answers (1)

Marek Šimůnek
Marek Šimůnek

Reputation: 86

ByteStringCoder is located in beam-sdks-java-extensions-protobuf module so you need to include dependency

https://mvnrepository.com/artifact/org.apache.beam/beam-sdks-java-extensions-protobuf

As for the TextIO: It uses StringUtf8Coder so you probably need to write your own BoundedSource/UnboundedSource. And then use:

pipeline.apply(Read.from(yourCreatedSource))

You could take inspiration of current TextSource, where you can probably only change type String of FileBasedSource for ByteString.

https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextSource.java

Upvotes: 1

Related Questions