CODEWITHSUNDEEP
apache-sparkpysparkapache-spark-sqldatabricks
Sarada Rout
Sarada Rout

Reputation: 57

DataFrame Lifespan in memory, Spark?

My question is more related to memory management and GC in sprak internally.

If I will create a RDD, how long it will leave in my Executor memory.

# Program Starts
spark = SparkSession.builder.appName("").master("yarn").getOrCreate()
df = spark.range(10)
df.show()
# other Operations
# Program end!!!
  1. Will it be automatically deleted once my Execution finishes. If Yes, Is there any way to delete it manually during program execution.
  2. How and when Garbage collection called in Spark. Can we implement custom GC like JAVA program and use it in Spark.

Upvotes: 0

Views: 1367

Answers (2)

Andrew Long
Andrew Long

Reputation: 933

"how long it will leave in my Executor memory."

In this particular case spark will no materialize the full dataset ever, instead it will iterate through one by one. Only a few operators materialize the full dataset. This includes, sorts/joins/groupbys/writes/etc

"Will it be automatically deleted once my Execution finishes."

spark automatically cleans any temp data.

"If Yes, Is there any way to delete it manually during program execution."

spark only keeps that data around if its in use or has been manually persisted. what are you trying to accomplish in particular?

"How and when Garbage collection called in Spark."

Spark runs on the JVM and the JVM with automatically GC when certain metrics are hit.

Upvotes: 1

vaquar khan
vaquar khan

Reputation: 11449

  1. DataFrame are Java objects so if no reference found your object is eligible to garbage collection
  1. Calling Custom gc not possible

Upvotes: 1

Related Questions