CODEWITHSUNDEEP
pythonapache-sparkpysparkapache-spark-sqljupyter-notebook
Edamame
Edamame

Reputation: 25416

Pyspark: spark data frame column width configuration in Jupyter Notebook

I have the following code in Jupyter Notebook:

import pandas as pd
pd.set_option('display.max_colwidth', 80)
my_df.select('field_1','field_2').show()

I want to increase the column width so I could see the full value of field_1 and field_2. I know we can use pd.set_option('display.max_colwidth', 80) for pandas data frame, but it doesn't seem to work for spark data frame.

Is there a way to increase the column width for the spark data frame like what we did for pandas data frame? Thanks!

Upvotes: 6

Views: 24721

Answers (4)

David
David

Reputation: 11593

I don't think you can set a specific width, but this will ensure your data is not cutoff no matter the size

my_df.select('field_1','field_2').show(10, truncate = False)

Upvotes: 20

abakar
abakar

Reputation: 301

you have just to add a 0 or False after the comma in show() , like below :

my_df.select('field1','field2').show(10,0) 
or
my_df.select('field1','field2').show(10,False)

Best,

Upvotes: 0

Manash Jyoti Das
Manash Jyoti Das

Reputation: 11

Both the following will work

my_df.select('field1','field2').show(10,False)

my_df.select('field1','field2').show(False)

Upvotes: 1

Andrey Vykhodtsev
Andrey Vykhodtsev

Reputation: 1061

This should give you what you want

import pandas as pd
pd.set_option('display.max_colwidth', 80)
my_df.select('field_1','field_2').limit(100).toPandas()

Upvotes: 1

Related Questions