CODEWITHSUNDEEP
apache-sparkapache-spark-sql
sai
sai

Reputation: 21

Why does "create table" yield an empty dataframe?

hc.sql("create table emp12(name String)");

res13: org.apache.spark.sql.DataFrame = []

scala> res13.printSchema
root

Why is the data frame empty but table is created in Hive when I am checking in Hive data warehouse?

hive> describe emp12;
OK
name                    string

Even when I load data from Spark, data is not going to Hive table.

Upvotes: 0

Views: 586

Answers (1)

zero323
zero323

Reputation: 330063

sql method returns the result of the query as a DataFrame so it makes sense only for SQL statements, which actually return any data. CREATE TABLE is just not one of these - it is a SQL (logical) command that is executed only for its side effect, which is registering a table in the catalog.

If you want to get the table you have issue separate query:

hc.sql("SELECT * FROM  emp12")

or just

hc.table("emp12")

Even when I load data from Spark, data is not going to Hive table.

That might be symptom of another problem, but it general you have to remember that Spark is no fully compatible with Hive, especially when using features like partitioning or bucketing.

Upvotes: 2

Related Questions