CODEWITHSUNDEEP
scalaapache-sparkdataframe
Feng Chen
Feng Chen

Reputation: 2253

How to display the results brough from column functions using spark/scala like what show() does to dataframe

I just started how to use dataframe and column in Spark/Scala. I know if I want to show something on the screen, I can just do like df.show() for that. But how can I do this to a column. For example, 

scala> val dfcol = df.apply("sgan")
dfcol: org.apache.spark.sql.Column = sgan

this can find a column called "sgan" from the dataframe df then give it to dfcol, so dfcol is a column. Then, if I do

scala> abs(dfcol)
res29: org.apache.spark.sql.Column = abs(sgan)

I just got the result shown on the screen like above. How can I show the result of this function on the screen like df.show() does? Or, in other words, how can I know the results of the functions like abs, min and so forth?

Upvotes: 2

Views: 2567

Answers (1)

Tzach Zohar
Tzach Zohar

Reputation: 37852

You should always use a dataframe, Column objects are not meant to be investigated this way. You can use select to create a dataframe with the column you're interested in, and then use show():

df.select(functions.abs(df("sgan"))).show()

Upvotes: 2

Related Questions