CODEWITHSUNDEEP
apache-sparkhiveqlapache-spark-sql
ggalka
ggalka

Reputation: 11

How to query Avro table in Spark SQL

I have key/value table in Hive 0.13 (key and value are avro types). When I try to run query in Spark SQL 1.1 I am getting following error (in Hive it works)

spark-sql> describe construct;
...
key struct<constructtype:string,...> from deserializer
value struct<...> from deserializer
...

spark-sql> select key.constructtype, count(*) from construct group by key.constructtype;
...

4/09/16 03:19:30 ERROR thriftserver.SparkSQLDriver: Failed in [select key.constructtype, count(*) from  construct group by key.constructtype]
org.apache.spark.sql.catalyst.errors.package$TreeNodeException: Unresolved attributes: 'key.constructtype,'key.constructtype, tree:
Aggregate ['key.constructtype], ['key.constructtype,COUNT(1) AS c_1#0L]

Upvotes: 1

Views: 847

Answers (1)

brosplit
brosplit

Reputation: 191

Check out the spark avro lib and see if this works:

https://github.com/databricks/spark-avro

Upvotes: 1

Related Questions