CODEWITHSUNDEEP
pysparkdatabricks
Chinmay Tornekar
Chinmay Tornekar

Reputation: 11

Dataframe display function in pyspark on databricks platform

I am new to databricks, i was studing topic dataframe in pyspark

df = spark.read.parquet(salesPath)
display(df)

Above is my code , i m not getting ,what actually the up arrows do? and why this beautiful df.display not included in Apache pyspark documentation?

pyspark-display-function-databricks

Upvotes: 0

Views: 295

Answers (1)

Alex Ott
Alex Ott

Reputation: 87329

Arrows are used to sort the displayed portion of the dataframe. But please note that the display function shows at max 1000 records, and won't load the whole dataset.

The display function isn't included into PySpark documentation because it's specific to Databricks. Similar function also exist in Jupyter that you can use with PySpark, but it's not part of the PySpark. (you can use df.show() function to display as text table - it's a part of the PySpark's DataFrame API)

Upvotes: 0

Related Questions