Best way to process a GCS file within Dataflow?

Question

I have a PCollection of matched GCS filenames, each of which contains a single compressed JSON blob. What's the best way to read the entire file, decompress it (Gzip format), and JSON decode it?

• TextIO is really close, but reads data per-line.
• GCS API offers an example for how to read the entire file, but it doesn't handle decompression and is leading me to reimplement a lot of core functionality.

Are there any existing APIs and/or examples that can give me a head start? Seems like this would be a pretty common use case.

Answer 1

This isn't natively supported in Dataflow. To accomplish reading a JSON blob out of a file, you could implement FileBasedSource:

https://cloud.google.com/dataflow/java-sdk/JavaDoc/com/google/cloud/dataflow/sdk/io/FileBasedSource

If that's enough to get started, we can continue to update this answer with more information.