CODEWITHSUNDEEP
scalaapache-sparkpysparkapache-spark-sql
Naresh
Naresh

Reputation: 5397

fetch more than 20 rows and display full value of column in spark-shell

I am using CassandraSQLContext from spark-shell to query data from Cassandra. So, I want to know two things one how to fetch more than 20 rows using CassandraSQLContext and second how do Id display the full value of column. As you can see below by default it append dots in the string values.

Code :

val csc = new CassandraSQLContext(sc)
csc.setKeyspace("KeySpace")
val maxDF = csc.sql("SQL_QUERY" )
maxDF.show

Output:

+--------------------+--------------------+-----------------+--------------------+
|                  id|               Col2|              Col3|                Col4| 
+--------------------+--------------------+-----------------+--------------------+
|8wzloRMrGpf8Q3bbk...|             Value1|                 X|                  K1|
|AxRfoHDjV1Fk18OqS...|             Value2|                 Y|                  K2|
|FpMVRlaHsEOcHyDgy...|             Value3|                 Z|                  K3|
|HERt8eFLRtKkiZndy...|             Value4|                 U|                  K4|
|nWOcbbbm8ZOjUSNfY...|             Value5|                 V|                  K5|

Upvotes: 41

Views: 106751

Answers (3)

Sairam Krish
Sairam Krish

Reputation: 11731

To show all rows of a dataframe, following can help :

df.show(n=df.count(), truncate=False)

Please be careful not to use this on a big dataFrame.

Quick reference on the parameters to show method (pyspark version):

Parameters
----------
n : int, optional
    Number of rows to show.
truncate : bool or int, optional
    If set to ``True``, truncate strings longer than 20 chars by default.
    If set to a number greater than one, truncates long strings to length ``truncate``
    and align cells right.

Upvotes: 2

eliasah
eliasah

Reputation: 40380

If you want to print the whole value of a column, in scala, you just need to set the argument truncate from the show method to false :

maxDf.show(false)

and if you wish to show more than 20 rows : 

// example showing 30 columns of 
// maxDf untruncated
maxDf.show(30, false)

For pyspark, you'll need to specify the argument name :

maxDF.show(truncate = False)

Upvotes: 71

WoodChopper
WoodChopper

Reputation: 4375

You won't get in nice tabular form instead it will be converted to scala object.

maxDF.take(50)

Upvotes: 6

Related Questions