CODEWITHSUNDEEP
apache-sparkhivepyspark
Alex R.
Alex R.

Reputation: 1437

Spark DataFrame vs sqlContext

For the purposes of comparison, suppose we have a table "T" with two columns "A","B". We also have a hiveContext operating in some HDFS database. We make a data frame:

In theory, which of the following is faster:

sqlContext.sql("SELECT A,SUM(B) FROM T GROUP BY A")

or

df.groupBy("A").sum("B")

where "df" is a dataframe referring to T. For these simple kinds of aggregate operations, is there any reason why one should prefer one method over the other?

Upvotes: 3

Views: 1274

Answers (2)

Rockie Yang
Rockie Yang

Reputation: 4925

Spark developers have made great effort to optimise. The performance between DataFrame Scala and DataFrame SQL is undistinguishable. Even for DataFrame Python, the differ is when collect data to driver.

It opens a new world

It doesn't have to be one vs. another

We can just choose what ever way we comfortable with

The performance comparison published by databricks enter image description here

Upvotes: 4

Justin Pihony
Justin Pihony

Reputation: 67075

No, these should boil down to the same execution plan. Underneath the Spark SQL engine is using the same optimization engine, the catalyst optimizer. You can always check this yourself by looking at the spark UI, or even calling explain on the resultant DataFrame.

Upvotes: 5

Related Questions