CODEWITHSUNDEEP
javaapache-sparkapache-spark-sqldatabricks
himanshu singh
himanshu singh

Reputation: 119

How to give Json schema file to spark 1.6 to load schema to spark DataFrame

Hi I'm Trying to load CSV file to spark dataframe. I'm using DataBricks CSV jar to load the data. I have data schema in a Json file and want to apply that schema to the DataFrame.

Below is My Json Schema File:-

 {
  "type" : "struct",
  "doc": "This is sample",
  "fields" : [ {
    "name" : "Name",
    "type" : "string" ,
    "nullable" : "true" 
  }, {
    "name" : "Address1",
    "type" : "string",
    "nullable" : "true" 
  }, {
    "name" : "Address2",
    "type" : "string",
    "nullable" : "true" 
  }, {
    "name" : "City",
    "type" : "string",
    "nullable" : "true" 
  }]
}

Upvotes: 0

Views: 580

Answers (1)

Srinu Babu
Srinu Babu

Reputation: 422

The following code may helpful for you.

StructType tempSchema = new StructType(new StructField[]{
            new StructField("name", DataTypes.StringType, true, Metadata.empty()),
            new StructField("Address1", DataTypes.StringType, true, Metadata.empty()),
            new StructField("Address2", DataTypes.StringType, true, Metadata.empty()),
            new StructField("City", DataTypes.StringType, true, Metadata.empty())
        });

    Dataset<Row> resultDs = spark.createDataFrame(dataRows, tempSchema);

Upvotes: 0

Related Questions