CODEWITHSUNDEEP
apache-sparkapache-spark-dataset
Crypto
Crypto

Reputation: 148

How to convert Java List<Date> to Spark Dataset<Row>

I have a list of dates that I want to convert to a spark dataset so that I can use the same in transformations.

List<Date> dates = new ArrayList<>();
dates.add(date1);

I expect something like this:

Dataset<Row> ds = sparkSession.createDataset(dates);

Upvotes: 2

Views: 2976

Answers (1)

Ramesh Maharjan
Ramesh Maharjan

Reputation: 41957

If you already have a list of Date then you can create a List of Row as

List<Row> data = new ArrayList<>();
for(Date date : dates) {
    data.add(RowFactory.create(date));
}

and then a schema as

StructType schema = new StructType(new StructField[] { 
                new StructField("date", DataTypes.DateType, false, Metadata.empty())
        });

and finally use the List of Row and schema to create the dataset as 

    Dataset<Row> ds = sparkSession.createDataFrame(data, schema);

which should give you a valid dataset

I hope the answer is helpful

Upvotes: 2

Related Questions