CODEWITHSUNDEEP
apache-sparkhiveavroavsc
Ajith Kannan
Ajith Kannan

Reputation: 842

hive external table on avro alias column not working

I tried creating hive table on avro using avsc spec file and need to renam some of the columns . used alias but seems its not working. the columns are returned as null when i query the table

SPARK DATAFRAME TO SAVE DATA

val data=Seq(("john","adams"),("john","smith"))
val columns = Seq("fname","lname")
import spark.sqlContext.implicits._
val df=data.toDF(columns:_*)
df.write.format("avro").save("/test")

AVSC Spec file

{
  "type" : "record",
  "name" : "test",
  "doc" : " import of test",
  "fields" : [  {
    "name" : "first_name",
    "type" : [ "null", "string" ],
    "default" : null,
    "aliases" : [ "fname" ],
    "columnName" : "fname",
    "sqlType" : "12"
  }, {
    "name" : "last_name",
    "type" : [ "null", "string" ],
    "default" : null,
    "aliases" : [ "lname" ],
    "columnName" : "lname",
    "sqlType" : "12"
  } ],
  "tableName" : "test"
}

EXTERNAL HIVE TABLE

create external table  test
STORED AS AVRO
LOCATION '/test'
TBLPROPERTIES ('avro.schema.url'='/test.avsc');

HIVE QUERY

SELECT last_name from test;

returns null even though there is data in avro with the original name ie lname

Upvotes: 3

Views: 255

Answers (0)

Related Questions