CODEWITHSUNDEEP
javahadoopapache-spark
jr.
jr.

Reputation: 45

JAVA - SPARK - saveAsTextFile can not be applied to '(java.lang.String, org.apache.hadoop.io.compress.CompressionCodec)'

Im writing a program in JAVA with Spark.

I have an JavaRDD named "copied_logs" which uses map and copies several fields from logs that are at on the hdfs.

now, I want to compress "copied_logs" with Bzip2 and then save it. i want to save this data on the hdfs using "saveAsTextFile" function. my code for compressing and saving is as follows:

    CompressionCodec codec = new BZip2Codec();
    copied_logs.saveAsTextFile(output_dir + "copied_logs.json", codec);

but i get this error:

Error:(128, 69) java: incompatible types: org.apache.hadoop.io.compress.CompressionCodec cannot be converted to java.lang.Class<? extends org.apache.hadoop.io.compress.CompressionCodec>

Thank you.

Upvotes: 2

Views: 897

Answers (1)

Markon
Markon

Reputation: 4600

The second argument of the method saveAsTextFile is a Class type (doc). Therefore, you need to pass the .class object.

Something like:

copied_logs.saveAsTextFile(output_dir + "copied_logs.json", BZip2Codec.class)

Upvotes: 2

Related Questions