CODEWITHSUNDEEP
javaapache-sparkapache-spark-mllib
user2736706
user2736706

Reputation: 103

How to create a Row from a List or Array in Spark using java

In Java, I use RowFactory.create() to create a Row:

Row row = RowFactory.create(record.getLong(1), record.getInt(2), record.getString(3));

where "record" is a record from a database, but I cannot know the length of "record" in advance, so I want to use a List or an Array to create the "row". In Scala, I can use Row.fromSeq() to create a Row from a List or an Array, but how can I achieve that in Java?

Upvotes: 9

Views: 28355

Answers (4)

Sanjay Singh
Sanjay Singh

Reputation: 1057

//Create a a list of DTO

List<MyDTO> dtoList = Arrays.asList(.....));

//Create a Dataset of DTO

Dataset<MyDTO> dtoSet = sparkSession.createDataset(dtoList,
                Encoders.bean(MyDTO.class));

//If you need dataset of Row

Dataset<Row> rowSet= dtoSet .select("col1","col2","col3");

Upvotes: 0

Alex Stanovsky
Alex Stanovsky

Reputation: 1366

For simple list values you can use Encoders:

 List<Row> rows = ImmutableList.of(RowFactory.create(new Timestamp(currentTime)));
 Dataset<Row> input = sparkSession.createDataFrame(rows, Encoders.TIMESTAMP().schema());

Upvotes: -1

Andrushenko Alexander
Andrushenko Alexander

Reputation: 1973

We often need to create Datasets or Dataframes in real world applications. Here is an example of how to create Rows and Dataset in a Java application:

// initialize first SQLContext
SQLContext sqlContext = ... 
StructType schemata = DataTypes.createStructType(
        new StructField[]{
                createStructField("NAME", StringType, false),
                createStructField("STRING_VALUE", StringType, false),
                createStructField("NUM_VALUE", IntegerType, false),
        });
Row r1 = RowFactory.create("name1", "value1", 1);
Row r2 = RowFactory.create("name2", "value2", 2);
List<Row> rowList = ImmutableList.of(r1, r2);
Dataset<Row> data = sqlContext.createDataFrame(rowList, schemata);

+-----+------------+---------+
| NAME|STRING_VALUE|NUM_VALUE|
+-----+------------+---------+
|name1|      value1|        1|
|name2|      value2|        2|
+-----+------------+---------+

Upvotes: 16

abaghel
abaghel

Reputation: 15297

I am not sure if I get your question correctly but you can use the RowFactory to create Row from ArrayList in java.

List<MyData> mlist = new ArrayList<MyData>();
    mlist.add(d1);
    mlist.add(d2);

Row row = RowFactory.create(mlist.toArray());

Upvotes: 12

Related Questions