CODEWITHSUNDEEP
scalaapache-sparkapache-spark-sqlapache-zeppelin
Softwaremaker
Softwaremaker

Reputation: 431

Remove Temporary Tables from Apache SQL Spark

I have registertemptable in Apache Spark using Zeppelin below:

val hvacText = sc.textFile("...")

case class Hvac(date: String, time: String, targettemp: Integer, actualtemp: Integer, buildingID: String)

val hvac = hvacText.map(s => s.split(",")).filter(s => s(0) != "Date").map(
    s => Hvac(s(0), 
            s(1),
            s(2).toInt,
            s(3).toInt,
            s(6))).toDF()

hvac.registerTempTable("hvac")

After I have done with my queries with this temp table, how do I remove it ?

I checked all docs and it seems I am getting nowhere.

Any guidance ?

Upvotes: 29

Views: 56943

Answers (4)

msrv499
msrv499

Reputation: 339

You can use sql drop table/view statement to remove it like below

spark.sql("drop view hvac");

Upvotes: 1

심형성 Hyung Sung Shim
심형성 Hyung Sung Shim

Reputation: 149

If you want to remove your temp table on zeppelin, try like this.

sqlc.dropTempTable("hvac")

or

%sql DROP VIEW hvac

And you can get the informations you need from spark API Docs(http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.package)

Upvotes: 14

zero323
zero323

Reputation: 330283

Spark 2.x

For temporary views you can use Catalog.dropTempView:

spark.catalog.dropTempView("df")

For global views you can use Catalog.dropGlobalTempView:

spark.catalog.dropGlobalTempView("df")

Both methods are safe to call if view doesn't exist and, since Spark 2.1, return boolean indicating if the operation succeed.

Spark 1.x

You can use SQLContext.dropTempTable:

scala.util.Try(sqlContext.dropTempTable("df"))

It can be still used in Spark 2.0, but delegates processing to Catalog.dropTempView and is safe to use if table doesn't exist.

Upvotes: 56

desaiankitb
desaiankitb

Reputation: 1052

in new ver (2.0 and latest) of spark. one should use: createOrReplaceTempView in place of registerTempTable (depricated) and corresponding method to deallocate is: dropTempView 

spark.catalog.dropTempView("temp_view_name") //drops the table

Upvotes: 12

Related Questions