CODEWITHSUNDEEP
amazon-s3apache-sparkdataframeorc
abstractKarshit
abstractKarshit

Reputation: 1465

Writing Spark dataframe in ORC format with Snappy compression

I am successfull in reading a text file stored in S3 and writing it back to S3 in ORC format using Spark dataframes. - inputDf.write().orc(outputPath);
What I am not able to do is convert to ORC format with snappy compression. I already tried giving option while writing as setting the codec to snappy but Spark is still writing as normal ORC. How to achieve writing in ORC format with Snappy compression to S3 using Spark Dataframes?

Upvotes: 4

Views: 4217

Answers (1)

abstractKarshit
abstractKarshit

Reputation: 1465

For anyone facing the same issue, in Spark 2.0 this is possible by default. The default compression format for ORC is set to snappy.

public class ConvertToOrc {
    public static void main(String[] args) {
        SparkSession spark = SparkSession
                .builder()
                .appName("OrcConvert")
                .getOrCreate();
        String inputPath = args[0];
        String outputPath = args[1];

        Dataset<Row> inputDf = spark.read().option("sep", "\001").option("quote", "'").csv(inputPath);
        inputDf.write().format("orc").save(outputPath);

   }
}

Upvotes: 3

Related Questions