CODEWITHSUNDEEP
apache-sparkpyspark
Arturo Sbr
Arturo Sbr

Reputation: 6333

Which storage levels are cleared by PySpark's `clearCahce()`?

Judging by the docs, it seems like spark.sql.Catalog.clearCache() only clears dataframes that are persisted in memory.

If I were to persist a table in disk (df.persist(StorageLevel.DISK_ONLY))), would cearCache() unpersist it too?

Upvotes: 0

Views: 58

Answers (1)

Jonathan
Jonathan

Reputation: 2043

In Spark, cache is one of the options for data persistence. clearCache() will not unpersist the data in your example, use unpersist(). It will marks the DataFrame as non-persistent, and remove all blocks for it from memory and disk.

Upvotes: 0

Related Questions