CODEWITHSUNDEEP
datasetschemaparquetapache-spark-2.0kryo
John Humanyun
John Humanyun

Reputation: 945

Writing parquet output with selected attributes from Bean

I have a bean class

@Getter
@Setter
public class Employee {
    String id;
    String name;
    String depart;
    String address;
    final String pipe= "|";

    @Override
    public String toString() {
        return id +pipe+ name +pipe+depart;
    }
}

And I have a JavaRDD<Employee> emprdd; and when I do the emprdd.saveAsText(path);. I get the output as based on the toString method. Now I wanted to write into the parquet format after converting it to the dataframe but I need only (id,name,depart). I tried sqlContext.createDataframe(rdd,Employee.class); (syntax ignored), but I dont need all the properties.

Can anyone guide me through this. (This is a sample , I have bean class with 350+ attributes)

Upvotes: 1

Views: 90

Answers (0)

Related Questions