Mihai Tache
Reputation: 171
how can you calculate the size of an apache spark data frame using pyspark?
Is there a way to calculate the size in bytes of an Apache spark Data Frame using pyspark?
Upvotes: 12
Views: 7318
Answers (1)
thePurplePython
Reputation: 2767
why don't you just cache the df and then look in the spark UI under storage and convert the units to bytes
df.cache()
Upvotes: 3
Related Questions
- How to find size (in MB) of dataframe in pyspark?
- How to find the size of a dataframe in pyspark
- How to find pyspark dataframe memory usage?
- How to estimate dataframe real size in pyspark?
- How can I estimate the size in bytes of each column in a Spark DataFrame?
- How to Determine The Partition Size in an Apache Spark Dataframe
- How to get the size of a data frame before doing the broadcast join in pyspark
- Count number of columns in pyspark Dataframe?
- How to get the size of an RDD in Pyspark?
- Calculate size of SparkR dataframe