CODEWITHSUNDEEP
hivepysparkapache-spark-sql
user1744585
user1744585

Reputation: 147

SparkSql get float type field value null from hive table

I create and import hive table with sqoop and use pyspark to get data. The table is composed by one string field, one int field and several float field. I can get the whole data by hue hive sql query. But while I program with pyspark sql the non-float field can be displayed and the float fields always show null value. HUE hive sql results: enter image description here

zeppelin pyspark output: enter image description here

The details of hive table:

enter image description here enter image description here

Upvotes: 0

Views: 888

Answers (1)

user1744585
user1744585

Reputation: 147

I finally found the cause. since I import these tables from mysql via sqoop. the original table columns are uppercase and in hive they were converted to all lowercase automatically. it caused all converted fields value can not be retrieved by sparksql. (but HUE hive queries these data normally, It might be a bug of spark.) I have to convert uppercase field names to lower case by specify the option --query in sqoop command. i.e. --query 'select MMM as mmm from table...'

Upvotes: 1

Related Questions