CODEWITHSUNDEEP
scalaapache-sparkapache-spark-sql
Yuri Brovman
Yuri Brovman

Reputation: 1113

Is there better way to display entire Spark SQL DataFrame?

I would like to display the entire Apache Spark SQL DataFrame with the Scala API. I can use the show() method:

myDataFrame.show(Int.MaxValue)

Is there a better way to display an entire DataFrame than using Int.MaxValue?

Upvotes: 45

Views: 132361

Answers (7)

Suresh G
Suresh G

Reputation: 147

Try with,

df.show(35, false)

It will display 35 rows and 35 column values with full values name.

Upvotes: 4

AkshayK
AkshayK

Reputation: 315

One way is using count() function to get the total number of records and use show(rdd.count()) .

Upvotes: 5

ayan guha
ayan guha

Reputation: 1257

As others suggested, printing out entire DF is bad idea. However, you can use df.rdd.foreachPartition(f) to print out partition-by-partition without flooding driver JVM (y using collect)

Upvotes: 2

Rajeev Rathor
Rajeev Rathor

Reputation: 1932

In java I have tried it with two ways. This is working perfectly for me:

1. 

data.show(SomeNo);

2. 

data.foreach(new ForeachFunction<Row>() {
                public void call(Row arg0) throws Exception {
                    System.out.println(arg0);
                }
            });

Upvotes: 0

keypoint
keypoint

Reputation: 2318

I've tried show() and it seems working sometimes. But sometimes not working, just give it a try:

println(df.show())

Upvotes: -3

Grega Kešpret
Grega Kešpret

Reputation: 12117

It is generally not advisable to display an entire DataFrame to stdout, because that means you need to pull the entire DataFrame (all of its values) to the driver (unless DataFrame is already local, which you can check with df.isLocal).

Unless you know ahead of time that the size of your dataset is sufficiently small so that driver JVM process has enough memory available to accommodate all values, it is not safe to do this. That's why DataFrame API's show() by default shows you only the first 20 rows.

You could use the df.collect which returns Array[T] and then iterate over each line and print it:

df.collect.foreach(println)

but you lose all formatting implemented in df.showString(numRows: Int) (that show() internally uses).

So no, I guess there is no better way.

Upvotes: 78

Justin Pihony
Justin Pihony

Reputation: 67115

Nothing more succinct than that, but if you want to avoid the Int.MaxValue, then you could use a collect and process it, or foreach. But, for a tabular format without much manual code, show is the best you can do.

Upvotes: 1

Related Questions