CODEWITHSUNDEEP
google-cloud-dataflowparquetapache-beam
Pari
Pari

Reputation: 1493

Example to read and write parquet file using ParquetIO through Apache Beam

Has anybody tried reading/writing Parquet file using Apache Beam. Support is added recently in version 2.5.0, hence not much documentation.

I am trying to read json input file and would like to write to parquet format.

Thanks in advance.

Upvotes: 4

Views: 8205

Answers (2)

Md Shihab Uddin
Md Shihab Uddin

Reputation: 561

Add the following dependency as ParquetIO in different module.

<dependency>
    <groupId>org.apache.beam</groupId>;
    <artifactId&gt;beam-sdks-java-io-parquet</artifactId>;
    <version>2.6.0</version>;
</dependency>;

//Here is code to read and write....

PCollection<JsonObject> input = #Your data
PCollection<GenericRecord> pgr =input.apply("parse json", ParDo.of(new DoFn<JsonObject, GenericRecord> {
        @ProcessElement
        public void processElement(ProcessContext context) {
            JsonObject json= context.getElement();
            GenericRecord record = #convert json to GenericRecord with schema
            context.output(record);
        }
    }));
pgr.apply(FileIO.<GenericRecord>write().via(ParquetIO.sink(schema)).to("path/to/save"));

PCollection<GenericRecord> data = pipeline.apply(
            ParquetIO.read(schema).from("path/to/read"));

Upvotes: 4

Nathan Nasser
Nathan Nasser

Reputation: 1004

You will need to use ParquetIO.Sink. It implements FileIO.

Upvotes: 3

Related Questions