CODEWITHSUNDEEP
javaapache-sparkapache-spark-sql
Malkolm
Malkolm

Reputation: 69

Spark : Named_struct requires at least one argument

I am mapping a Dataset of Row to a Dataset of a custom class.

Dataset<Row> rows= sparkSession.read().orc(path);
Dataset<customClass> dataset =
  rows.map(I'm parsing row to map it to customClass, 
    Encoders.bean(customClass.class));

And I am getting this AnalysisException:

AnalysisException: cannot resolve 'named_struct()' due to data type mismatch: input to function named_struct requires at least one argument;

I am using Spark 2.3.0 and am encoding my custom class using javaBeans.

I checked whether the schema was effectivly inferred by Encoders and it is the case. So, technically, the map operation should work.

Has anyone ever faced this exception message ? What does the named_struct function do ? I found no relevant information related to Spark ...

root
 |-- field1: struct (nullable = true)
 |    |-- value: string (nullable = true)
 |-- field2: string (nullable = true)
 |-- field3: integer (nullable = true)
 |-- field4: double (nullable = true)
 |-- field5: struct (nullable = true)
 |    |-- value: double (nullable = true)
 |-- field6: struct (nullable = true)
 |    |-- field61: double (nullable = true)
 |    |-- field62: string (nullable = true)
 |    |-- field63: integer (nullable = true)
 |    |-- field64: struct (nullable = true)
 |    |    |-- value: string (nullable = true)
 |-- field7: struct (nullable = true)
 |    |-- value: double (nullable = true)
 |-- field8: struct (nullable = true)
 |    |-- value: double (nullable = true)
 |-- field9: struct (nullable = true)
 |    |-- field91: map (nullable = true)
 |    |    |-- key: struct
 |    |    |-- value: struct (valueContainsNull = true)
 |    |    |    |-- value: string (nullable = true)
 |    |    |    |-- field911: struct (nullable = true)
 |    |    |    |    |-- value: double (nullable = true)
 |    |    |    |-- field912: struct (nullable = true)
 |    |    |    |    |-- value: double (nullable = true)
 |    |    |    |-- field913: map (nullable = true)
 |    |    |    |    |-- key: struct
 |    |    |    |    |-- value: struct (valueContainsNull = true)
 |    |    |    |    |    |-- value: integer (nullable = false)
 |    |    |    |    |    |-- field9131: struct (nullable = true)
 |    |    |    |    |    |    |-- value: double (nullable = true)
 |    |    |    |    |    |-- field9131: struct (nullable = true)
 |    |    |    |    |    |    |-- value: double (nullable = true)
 |    |    |    |-- field914: struct (nullable = true)
 |    |    |    |    |-- value: double (nullable = true)
 |    |    |    |-- field915: string (nullable = true)
 |-- field10: string (nullable = true)
 |-- field11: struct (nullable = true)
 |    |-- field111: map (nullable = true)
 |    |    |-- key: struct
 |    |    |-- value: struct (valueContainsNull = true)
 |    |    |    |-- value: integer (nullable = false)
 |    |    |    |-- field1111: struct (nullable = true)
 |    |    |    |    |-- value: double (nullable = true)
 |    |    |    |-- field1112: struct (nullable = true)
 |    |    |    |    |-- value: double (nullable = true)
 |-- field12: boolean (nullable = true)
 |-- field13: struct (nullable = true)
 |    |-- field131: integer (nullable = false)
 |    |-- field132: integer (nullable = false)
 |-- field14: struct (nullable = true)
 |    |-- field141: string (nullable = true)

Upvotes: 1

Views: 2659

Answers (1)

Malkolm
Malkolm

Reputation: 69

I finally found why I had this named_struct error : One of the fields I was using was declared final, which means it didn't have a setter. This violates the JavaBean contract.

Upvotes: 3

Related Questions